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INTRODUCTION 

The  Subject 
The  controversy  surrounding  the  procurement  of  the  C-5A  Galaxy, 

! 

a  new  transport  aircraft  obtained  for  the  U.S.  Air  Force,  has  become 
a  cause  celebre   for  Congressional  and  Public  critics  of  Department  of 
Defense  spending.  The  occurrence  of  exceptional  cost  over-runs  in 

this  program  has  been  cited  as  evidence  of  poor  management  practice. 

I 
One  cannot  refute  this  contention,  for  great  excesses  in  spending 

I 
above  budgeted  levels  per  force  demonstrate  poor  management.  However, 

it  is  important  to  determine  which  management  techniques  have  failed, 
if  circumstances  such  as  those  experienced  in  the  Galaxy  program  are 
to  be  avoided  in  the  future. 

There  is  little  doubt  that  the  Galaxy  program  stretched  the 
state-of-the-art,  to  a  degree  not  originally  anticipated,  in  aero- 
nautical engineering  and  manufacturing,  when  it  sought  to  reach  new 
technological  plateaus.   It  is  now  evident  that  economic,  as  well  as 
technological  factors,  contributed  to  unforseen  developments.  The 

initial  planning  estimates,  made  in  1964  and  based  on  a  preceding 

•  ■  ! 

five-year  period  marked  by  a  relatively  stable  economic  environment, 

j 
would  certainly  have  been  revised  upward  if  conditions  experienced 

during  the  succeeding  five  years  had  been  anticipated. 

Though  the  Galaxy  program  did  experience  certain  technolog- 
ical and  economic  consequences  which  might  be  regarded  as  extraordinary, 


it  has  not  been  unique  in  terms  of  unanticipated  expenditures.   It 
has  been  observed  that  "initial  cost  estimates  of  major  weapons  sys- 
tems have  historically  escalated,  just  as  they  have  in  the  C-5A." 

One  should  not  conclude  from  the  preceding  that  problems  in 
controlling  cost  increases  are  peculiar  to  government.   Much  of  the 

unexpected  expense  for  the  Galaxy  had  to  be  absorbed  by  Lockheed  Air- 

j 
craft  Corporation,  the  prime  contractor,  since  initial  contracts,  based 

on  Lockheed  estimates,  did  not  provide  for  the  expenditures  that  were 

i 
required.   Corporations  other  than  Lockheed  have  also  experienced  simi- 
lar predicaments.   For  example,  General  Dynamics,  while  still  trying  to 
recover  from  the  financial  calamities  experienced  with  the  F-lll  mili- 
tary aircraft  and  the  Convair  880-990  commercial  jets,  is  now  encoun- 

i 
2     ! 
tering  losses  in  shipbuilding  programs;  all  due  to  unplanned  cost. 

It  is  possible  to  gain  some  insight  into  the  difficulties  that 

have  been  described  if  recognition  is  given  to  the  commonality  which  is 

displayed:   each  of  the  ventures  which  has  been  cited  relied  heavily  on 

research  and  development  to  provide  techniques  and  components  needed 

j 
for  the  achievement  of  program  objectives.   This  fact  would  then  lead 

! 
one  to  conclude  that  a  major  contributor  to  problems  of  this  type  is  a 

growing  cost  of  technological  advance. 

Improvements  in  technology  derive  chiefly  from  research  and 

development  efforts.   It  has  been  observed,  in  this  regard,  that  as 

technology  does  advance,  the  scientific  disciplines,  to  be  considered 


Armed  Forces  Management,  (July,  1969),  p.  58. 

2"General  Dynamics:   In  Trouble  Again,"  Business  Week,  (October 
4,  1969),  pp.  48-52. 


in  seeking  continued  advance,  become  more  esoteric  in  nature  and  great- 
er  in  number.   Thus,  as  program  objectives  move  to  successively  higher 


levels  of  sophistication,  the  number  of  alternatives  available  for 
realization  of  the  objectives  increases,  as  does  the  costs  of  the 
alternatives. 

The  problem,  then,  is  one  which  has  increased  in  significance 
in  the  face  of  technological  advances,  and  derives  from  difficulties 
encountered  in  both  selecting  appropriate  courses  of  action  for  a- 
chieving  objectives  and  in  properly  forecasting  and  controlling  the 
expense  to  be  realized  in  pursuing  the  selected  course.   It  has  been 
recognized  that  there  is  a  growing  need  for  improved  techniques  to  be 
made  available  to  managers  for  the  purpose  of  making  hard  choices  in 
the  funding  of  research  and  development  efforts. 

The  Objective 
A  major  facet  of  the  research  and  development  project  man- 
ager's role  is  essentially  one  of  control.  This  is  exercised  by 
monitoring  project  progress  and  ensuring  that  achievement  and  asso- 
ciated expenditures  are  kept  within  planned  limits.  This  function 
will  also  include  the  task  of  affecting  certain  modifications,  as 
necessary,  to  .correct  deviations  from  the  plan. 


Marvin  J.  Cetron,  "Technological  Forecasting:  A  Prescrip- 
tion for  the  Military  R  S  D  Manager,"  Naval  War  College  Review, 
XXI  (April,  1969),  p.  14. 

William  E.  Souder,  "Experiences  with  an  R  5  D  Project  Con- 
trol Model,"  IEEE  Transactions  on  Engineering  Management,  EM-15 
(March,  1968),  p.  40. 


Control  of  research  and  development  is,  of  course,  closely  re- 
lated to  the  planning  function  which  entails  the  selection  of  pro- 
grams and  the  scheduling  and  allocation  of  funds.   Control  and  plan- 
ning interact  principly  when  information,  obtained  through  control, 

I 
causes  changes  in  program  scheduling  or  fund  allocation.   For  instance, 


the  progress  realized  in  a  program  may  result  in  that  program  being 
terminated  entirely,  or  being  expanded  at  the  expense  of  some  other 
program. 

Difficulties  are  encountered  in  research  and  development 
management  which  are  due  to  an  intrinsic  uncertainty.  Frequently  it 
is  impossible  to  discern  whether  a  program  is  on  the  verge  of  failure 
or  success.   This  uncertainty  is  also  quite  prevelant  in  program  se- 
lection since  a  priori   estimates  are  often  little  more  than  educated 
guesses,  and  the  selection  of  programs  on  the  basis  of  merit  is  often 
difficult. 

The  objective  of  this  paper  is  to  develop  a  technique  for 

evaluating  the  relative  value  of  proposed  programs,  so  as  to  assist 

i 

in  program  selection;  for  monitoring  program  progress  and  evaluating 

! 

observed  deviations;  and  for  providing  information  to  be  used  in  ad- 

: 

justment  of  original  planned  program  content  and  expenditure  levels. 

.  .  I 

Proposed  programs  are  to  be  assessed  on  the  basis  of  some  selected 

I 
measure  of  utility  so  that  available  funds  may  be  allocated  among 

alternative  proposals  in  order  to  realize  a  maximum  return.   Program 

i 
progress  is  to  be  measured  in  terms  of  achievement  versus  cost  and 

achievement  versus  time.  Thus,  it  will  be  possible  to  identify  pro- 
grams that  either  fail  to  meet  prescribed  time  schedules  or  exceed 


budgeted  expenditures.   Finally,  using  data  obtained  from  the  analysis 
of  program  progress,  it  will  be  possible  to  recommend  changes  in  pro- 
gram mix  in  keeping  with  observed  results. 

As  was  stated,  it  is  intended  that  the  technique,  which  is  to 
be  developed,  will  be  useful  only  in  selecting  from  among  proposed 

programs.   It  is  not  intended  to  formulate  proposals.  The  actual 

l 

formulation  of  proposals  must  be  done  on  the  basis  of  objectives  and 
applicable  policy  considerations.  This  requires  a  certain  amount  of 
subjective  evaluation.  Only  after  this  evaluation  has  been  completed, 
and  acceptable  candidate  programs  selected,  is  it  possible  to  conduct. 
an  objective  analysis  on  the  basis  of  the  parameters  mentioned  above. 

In  order  to  facilitate  the  development  of  an  analytic  technique, 
it  will  be  necessary  to  make  certain  assumptions  regarding  the  struc- 
ture of  subject  programs,  the  statistical  independence  of  programs 
and* the  probabilistic  character  of  research  and  development  efforts. 
However,  necessary  assumptions  will  be  examined  to  ascertain  the 
significance  of  any  inaccuracies  which  may  be  introduced. 

The  technique  which  is  to  be  generated  will  derive  from  man- 
agement systems  common  to  Department  of  Defense  research  laboratories. 
As  a  result,  it  will  not  be  universally  applicable  in  all  management 
situations.  However,  since  the  intent  is  to  simply  demonstrate  a 
means  of  incorporating  certain  analytic  methods  into  the  management 

process,  the  methods  themselves  should  be  easily  adaptable  to  other 

i 
management  systems.  Throughout  the  ensuing  analysis,  appropriate 

j 
-  r 
analogy  will  be  made,  as  required,  to  assist  in  relating  the  dis- 
cussion to  non-defense  oriented  circumstances. 


Definitions 
This  paper  constitutes  a  merger  of  several  disciplines  which 
are  of  different  nature  and  utilize  a  variety  of  nomenclatures.  Since 
the  discussion  is  foremost  intended  as  a  business-oriented  work  it  is 
assumed  that  the  reader  is  familiar  with  the  terminology  associated 
with  management  science  and  economics.  Moreover,  due  to  the  essential 
quantitative  character  of  the  paper,  it  must  be  presumed  that  there  is 
also  some  familiarity  with  the  semantic  peculiarities  of  mathematics; 
particularly  probability  theory  and  statistical  analysis.  However, 
certain  terms  which  lack  precise  meaning,  and  are  used  frequently  in 
this  paper,  do  deserve  definition  as  follows: 

1.  research  and  development. --activities  which  seek  to  advance 
the  state-of-the-art  in  any  technological  discipline.  The  term  research 
is  usually  associated  with  those  activities  concerned  with  basic  con- 
cepts and  theories  and  which  devote  considerable  effort  to  identifying 

! 
and  defining  topics  not  previously  studied.   Development,  on  the  other 

! 

hand,  is  generally  considered  to  include  those  activities  directed  at 

| 
refining  and  extending  knowledge  of  existing  topics.  Research  and 

i 
development  are  considered  together  since  research  prompts  development 

I 
and  since  both  are  endeavors  which,  presumably,  have  not  previously 

been  conducted. 

2.  project. — any  organized  research  and  development  activity, 

conducted  by  a  single  group  and  having  defined  objectives.  A  project 

is  the  lowest  level  of  research  and  development  organization,  and  can- 

I 

not  be  decomposed  into  other  activities  which  are  capable  of  satis-   .  : 
-fying_proj  ect_obj  ect :ives_. , ■ j 


3.  program. --a  research  and  development  effort  composed  of 
several  projects.  A  necessary  characteristic  of  a  program  is  that  its 

i 

objectives  can  be  satisfied  even  if  one  or  more  of  its  constituent 
parts,  the  projects,  are  discontinued. 

The  Procedure 

Chapter  I,  "A  Review  of  Current  Methods,"  presents  a  survey 

of  efforts  by  other  authors  in  developing  analytic  procedures  for  use 

J  in  planning  and  controlling  research  and  development  programs.  The 
i 

|  purpose  of  this  survey  is  to  describe  the  attempts  that  have  been  made 
j 

in  the  past  and  to  evaluate  the  degree  of  success  that  has  been  real- 
ized for  the  purpose  of  identifying  difficulties  which  have  been  en- 
'  countered.  This  survey  is  also  intended  to  provide  a  general  intro- 
duction to  the  elements  of  quantitative  management  methods. 

Chapter  II,  "Description  of  the  Technique,"  examines  in  depth 
the  two  principle  components  of  quantitative  research  and  development 
management  systems;  project  selection  and  control.  This  description 
is  in  the  form  of  an  examination  of  various  techniques  that  have  been 
developed  in  the  past  for  use  in  selection  and  control.  This  chapter 
forms  the  foundation  for  the  techniques  to  be  originated  in  this  paper. 

Chapter  III,  "Construction  of  the  Model,"  presents  the  deriva- 
tion of  the  selection  and  control  techniques  which  are  the  subject  of 
the  paper.  The  derivation  demonstrates  a  method  of  organizing  elements 
of  the  procedures  discussed  in  Chapter  II  such  that  an  improved  ap- 
proach to  the  problem  of  project  selection  is  obtained.  Also,  a 
technique  for  optimizing  project  selection  is  presented.  The  chapter 


8 

concludes  with  the  development  of  a  method  for  controlling  research 
and  development  projects  that  is  based  on  statistical  inference. 

Chapter  IV,  "Conclusions,"  summarizes  the  status  of  quantita- 
tive selection  and  control  techniques  as  it  stands  at  present,  and 
reviews  the  role  that  is  anticipated  for  the  techniques  offered  in 
Chapter  III.   In  addition,  elements  of  the  subject  techniques  which 
deserve  further  refinement  are  discussed. 

Appended  to  the  text,  in  Appendix  A,  is  a  sample  of  mat- 
aerials  which  are  considered  useful  in  the  implementation  of  quanti- 
tative selection  methods.   Appendix  B  presents  a  tabulation  of  the 
cumulative  Poisson  distribution  which  is  used  in  the  control  technique 
of  Chapter  III.  Finally,  Appendix  C  presents  a  partial  tabulation  of 
the  chi-square  distribution,  also  referred  to  in  Chapter  III. 


CHAPTER  I 
A  REVIEW  OF  CURRENT  METHODS 

Focusing  On  the  Problem 

As  described  in  the  introduction,  the  problem  under  considera- 
tion is  one  of  developing  more  effective  methods  for  the  management  of 
research  and  development  programs.  A  specific  provision  was  included 
which  precludes  processes  involved  in  the  formulation  of  programs. 
These  processes,  which  are  generally  classified  under  the  heading  of 
normative  forecasting,  have  to  do  with  the  identification  of  goals  and 
selection  of  candidate  programs  on  the  basis  of  policy  criteria;  usual- 
ly excluding  considerations  of  economy  and  technological  feasibility. 

After  the  normative  forecasting  process  is  complete,  and  a  set 
of  acceptable  research  programs  has  been  generated,  management  then 
tries  to  determine  which  programs  from  the  set  to  pursue  (project  se- 
lection), how  much  money  to  spend  on  each  program  (budget  allocation), 
and  the  size  of  the  total  budget  for  all  the  programs  selected  (budget 
determination).   The  first  two  of  these  functions,  project  selection 
and  budget  allocation,  constitute  the  key  areas  of  interest  for  this 
paper. 

As  was  stated  earlier,  project  selection  is  essentially  a 


1E.  M.  Rosen  and  W.  E.  Souder,  "A  Method  for  Allocating  R  5  D 
Expenditures,"  IEEE  Transactions  On  Engineering  Management,  EM-12 
(September,  1965),  p.  90. 
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1 

planning  activity.  However,  budget  allocation  is  a  planning  problem 
also;  it  takes  place  at  the  start  of  a  program  or  at  a  review  point 
where  some  change  of  plans  is  indicated.  The  key  decisions  in  budget 

! 

(or  resource)  allocation  involve  the  rates  of  resource  expenditure  and 

I 

the  planned  dates  for  the  commencement  of  an  activity. 

'  It  should  be  re-iterated  that,  since  attention  is  going  to  fo- 

I 
cus  on  planning  and  plan  review,  consideration  is  appropriately  due 

I 
ithe  associated  control  processes.   In  this  respect,  examination  of 

!  | 

jmanagement  processes  will  include  those  concerned  with  monitoring  pro- 

I 
gram  progress  and  ensuring  that  prescribed  limits  are  observed. 


The  Analytic,  Framework 

In  the  abstract,  one  can  consider  research  as  simply  the  pur- 

! chase  of  information  upon  which  to  base  later  and  better  decisions.  All 

(but  one  of  these  decisions  is  whether  or  not  to  continue  purchasing  in- 

I 

'formation  (that  is  research).  The  one  exception  is  the  final  decision, 

I  which  is  whether  or  not  to  commercialize  the  results.  The  decisions  to 

continue  a  research  effort,  and  the  subsequent  decisions  concerned  with 

how  to  conduct  the  research  do  ?ppear  to  be  sequential  in  nature.  This 

sequential  decision  making  process  has  been  the  subject  of  descriptive 

2 
work  in  the  past.   It  is  worthwhile  to  note  that  the  applicability  of 


R.  S.  Rosenbloom,  "Notes  on  the  Development  of  Network  Models 
for  Resource  Allocation  in  R  5  D  Projects,"  IEEE  Transactions  on  Engi- 
neering Management,  EM- 11  (June,  1964),  p.  62. 


D.  L.  Marples,  "The  Decisions  of  Engineering  Design,"  IRE 
Transactions  on  Engineering  Management,  EM-8  (June,  1961),  pp.  55-71. 
and  T.  A.  Marshak,  "Strategy  and  Organization  in  a  System  Development 
Project,"  The  Rate  and  Direction  of  Inventive  Activity  (Princeton,  New 
Jersey:   Princeton  University  Press,  1962),  pp.  461-475. 
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the  sequential  decision  model  for  research  efforts  is  a  function  of 
the  degree  of  structure  that  is  evidenced  in  the  effort.   For  example, 
in  basic  research,  which  is  fairly  unstructured,  the  model  should  be 
quite  appropriate.   In  developmental  design,  however,  decisions  to 
commence  a  phase  of  the  program  may  be  made  before  preceding  phases 
are  complete.   In  this  case,  then,  some  inaccuracies  may  be  intro- 
duced. Nonetheless,  for  the  pruposes  of  the  present  discussion,  the 
model  can  be  presumed  adequate. 

The  concept  of  sequential  decisions  is  only  one  ingredient  in 
a  characterization  of  research  and  development  processes.   Previous 
studies,  for  instance,  have  identified  certain  parameters  as  being 
significant  in  research  efforts.  Among  these  are  the  magnitude  of 
total  expenditures,  duration,  work  scope,  activity  content,  geographic 
dispersion  of  related  elements,  and  others.   Furthermore,  any  one  or 
more  of  these  parameters  may  be  selected  as  a  primary  measure;  de- 
pending on  the  type  and  objectives  of  the  analysis  for  which  the 
characterization  is  intended.  Of  the  numerous  parameters  which  might 
be  used,  certain  ones  have  been  found  to  be  more  useful  as  constituents 
of  a  model  for  the  research  process.  These  are: 

1)  the  items  worked  on, 

2)  the  activities  (operations)  performed  on  these  items, 

3)  the  duration  of  each  assignment, 

4)  the  effort  (or  resources)  utilized  by  each  assignment. 

Furthermore,  the  dynamics  of  the  research  effort  have  been  found  to  be 
adequately  represented  by  the  following: 

1)  the  start  and  stop  time  of  each  assignment,  in  calendar 
time, 

2)  the  total  effort  content  of  each  assignment. 
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3)  the  distribution  of  this  effort  content  over  the  duration 
of  the  assignment. 

A  final  aspect  of  the  research  process  which  should  be  con- 
sidered in  an  analytic  characterization  is  the  representation  of  the 
economies  of  operation.   Here,  economy  of  operation  relates  to  the  rate 
at  which  resources  can  be  effectively  employed.   In  this  respect,  the 
research  process  is  partitioned  into  three  funding  stages.  The  first 
stage  lies  below  a  "critical"  cost  level,  such  that  any  allocation  be- 
low  this  level  yields  no  return.  Above  the  critical  cost  level  is  a 
! stage,  within  which,  progress  is  a  monotonically  increasing  function 

i 

i of  cost.   This  mid-stage,  which  can  be  called  the  "adequate"  stage, 
[has  an  upper  termination  at  a  funding  level  called  the  "satiation" 
i  cost.  Above  this  cost  level  increases  in  cost  have  no  corresponding 

!•         •  * 

I  increases  in  progress. 

In  concluding  this  discussion  of  the  analytic  approach  to  re- 

- 
| search  and  development  planning  and  control,  it  is  considered  that  the 

interactions  of  the  several  components  of  the  management  system  can  be 
summarized  by  graphic  display  as  shown  in  Figure  1.   It  is  intended 
that  this  should  illustrate  the  inter-related  nature  of  planning  and 
control,  and  indicate  the  fashion  in  which  each  associates  with  pro- 
gram operation. 


P.  V.  Norden,  "On  the  Anatomy  of  Development  Projects,"  IRE 
Transaction  on  Engineering  Management,  EM- 7  (March,  1960),  p.  37. 


R.  J.  Freeman,  "A  Stochastic  Model  for  Determining  the  Size 
and  Allocation  of  the  Research  Budget,"  IRE  Transactions  on  Engineering 
Management ,  EM-7  (March,  1960),  p.  4. 
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Some  Current  Techniques 
Numerous  authors  have  offered  quantitative  techniques  for  the 


management  of  research  and  development.   Of  these,  the  techniques 

t 

that  have  objectives  similar  to  those  of  this  paoer  are  described  below: 

i 

.  1)  Asher,  1962, --This  method  was  derived  for  the  purpose  of    ; 

i 
man-power  allocation  in  a  pharmaceutical  firm.  The  method  considers  a 

i 

number  of  alternative  projects  on  the  basis  of  discounted  net  value,  if 
successful;  probability  of  success;  and  man-hours  (resources)  required, 
including  skill  levels.   From  these  data  an  expected  discounted  value 
is  computed.  Next,  linear  programming  is  used  to  assign  various  re- 
search teams,  characterized  by  skill  levels  represented,  to  maximize 

the  expected  return  subject  to  a  constraining  number  of  hours  available 

2  i 

for  each  team. 

2)  Atkinson  and  Bobis,  1969. — This  is  a  method  for  determin- 
ing money  to  be  spent  on  product  oriented  research  programs.  The 
method  commences  by  estimating  the  density  function  of  the  probability 
of  a  project  succeeding  for  a  given  expenditure.  This  distribution 
is  given  in  the  form  of  a  logistic  function.  On  the  basis  of  this 
function,  the  probability  of  completing  a  program  within  a  given  time, 
for  a  given  expenditure,  is  determined. 


*For  a  brief  but  thorough  review  of  current  techniques,  in- 
cluding an  exceptional  bibliography  of  the  topic  see  M.  J.  Cetron,  J. 
Martino  and  L.  Roepcke,  "The  Selection  of  R  5  D  Program  Content  —  Survey 
6f  Quantitative  Methods,"  IEEE  Transactions  on  Engineering  Management, 
EM-14  (March,  1967),  pp.  4-13. 

2 

D.  T.  Asher,  "A  Linear  Programming  Model  for  the  Allocation  of 
R  and  D  Efforts,"  IRE  Transactions  on  Engineering  Management,  EM-9 
(December,  1962),  pp.  154-57. 
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The  next  step  in  the  analysis  involves  a  determination  of  com- 
mercial value  as  a  function  of  completion  time.  This,  in  conjunction 
with  estimates  of  completion  time  yields  an  expected  commercial  value. 
Having  these  data,  it  is  then  possible  to  optimize  the  expected  value 
of  research  on  the  basis  of  annual  expenditures  for  the  alternative 
j  programs  being  considered.   Due  to  the  formidable  nature  of  this  op- 
!  timization  problem  which  results  from  the  numerous  variables  involved, 
an  iterative  optimization  technique  is  employed. 


3)  Dean  and  Hauser,  1967. --The  employment  of  this  method  com- 

j  mences  with  a  decomposition  of  the  research  program  into  the  following 

;  catagories : 
i 

a)  Qualitative  Materiel  Development  Objectives  (QMDO)--the 

several  major  goals  of  the  research  (effort);  corresponding  approximate 
ly  to  program  objectives  as  defined  in  the  Introduction. 

b)  Materiel  Concepts—the  various  different  component  efforts 
of  the  QMDO  (for  instance,  if  a  QMDO  is  the  development  of  a  missile, 

i  these  might  be  guidance,  propulsion  and  structure);  corresponding 

i 

j  approximately  to  program  content. 

c)  Technical  Approaches — the  several  alternative  methods  of 
achieving  a  materiel  concept;  corresponding  to  different  projects 
which  satisfy  the  same  objectives. 

The  utility  analysis  of  the  research  program  is  affected  by 
first  estimating  the  probability  of  success  and  cost  for  the  technical 


A.  C.  Atkinson  and  A.  H.  Bobis,  "A  Mathematical  Basis  for  the 
Selection  of  Research  Projects,"  IEEE  Transactions  on  Engineering 
Management,  EM- 16  (February,  1969),  pp.  2-8. 
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approaches.  With  these  data  it  is  then  possible  to  determine  the  al- 
location of  funds,  among  the  technical  approaches  for  a  given  material 
concept,  which  maximizes  the  probability  of  the  material  concept  suc- 
ceeding; subject  to  the  constraint  of  a  fixed  amount  being  available 
for  the  material  concept  funding. 

Assuming,  that,  in  order  for  a  QMDO  to  succeed,  all  component 
materiel  concepts  must  succeed,  it  is  then  possible  to  find  the  fund- 
ing levels  for  materiel  concepts  in  order  to  maximize  the  QMDO  proba- 
bility of  success;  subject  to  a  budget  constraint. 

This  particular  method  has  been  found  amenable  to  optimization 
j 
by  dynamic  programming.   The  authors  maintain  that  such  a  solution 

■contributes  flexibility  and  speed  to  solution,  and  is  particularly 

'  suitable  for  the  type  of  structure  used  in  decomposition  of  the 
i 

j program. * 

4)  Hess,  1962. --Hess  introduced  the  first  dynamic  programming 

| solution  to  the  allocation  problem.   His  solution  assumes  values  for 
certain  parameters,  among  them  being  the  probability  of  technical 
success,  expected  profit  as  a  function  of  time  of  completion,  and  an 
assumed  measure  of  usefulness  deriving  from  prior  research.  Using  the 
dynamic  technique  that  is  developed,  it  is  possible  to  obtain  an  op- 
timal allocation  of  funds  among  programs  and  budgeted  rates  of  ex- 
penditure for  each  program;  subject  to  appropriate  overall  budget 


1B.  V.  Dean  and  L.  E.  Hauser,  "Advanced  Materiel  Systems  Plan- 
ning," IEEE  Transactions  on  Engineering  Management,  EM- 14  (March,  1967) 
pp.  21-43. 


17 


Aspects  of  Uncertainty 
Complications  attendent  to  uncertainty  arising  from  opera- 
tional situations  are  by  no  means  limited  to  management  of  research 
and  development  activities.   Similar  problems  are  common  to  the 
management  of  most  business  efforts.  This  is  evidenced  by  the  de- 
jvelopment  of  disciplines  such  as  statistical  decision  theory,  and 

! their  rise  to  prominence  in  management  science.   Research  efforts, 

! 

j however,  are  somewhat  unique  in  that  they  demonstrate  a  degree  of 

I  endogenous  uncertainty  not  characteristic  of  other  management  pro- 

! cesses.  Even  if  environmental  factors  (such  as  market  conditions  and 
i 

| the  supply  of  materials,  labor  and  funds)  were  deterministic,  the  re- 
i suits  of  research  and  development  programs  would  not  be.  Thus,  the 
significance  of  uncertainty  in  research  management  has  added  impor- 
tance. 

As  is  indicated  by  the  preceding  description  of  some  available 
techniques,  this  uncertainty  is  usually  accounted  for  by  means  of  a 
"probability  of  success"  term.  This  approach  to  the  problem,  however, 
has  some  disadvantages. 

To  begin  with,  there  is  no  universally  accepted  meaning  of 
probability  of  success.  Most  authors  account  for  this,  to  some  extent, 
by  allowing  for  a  probability  of  technical  success  and,  also,  for 


IS.  W.  Hess,  "A  Dynamic  Programming  Approach  to  R  and  D  Budg- 
eting and  Project  Selection,"  IRE  Transactions  on  Engineering  Manage- 
ment, EM- 9  (December,  1962),  pp.  170-79T 
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commercial  success.   Even  if  one  were  to  limit  consideration  to 
technical  success  alone,  a  probability  measure  is  inadequate,  since  it 
does  not  allow  for  varying  degrees  of  success  which  might  result.   In 
reference  to  the  probability  of  success  concept,  Baker  and  Pound  ob- 
served that  "at  best,  it  is  extremely  difficult  to  understand  what  is 
meant  by  such  a  term."* 

The  use  of  probability  of  success  is  further  complicated  by 
the  fact  that  it  is  extremely  difficult  to  estimate  a  value  for  such  a 

parameter.   In  the  work  done  by  Asher,  for  instance,  probabilities 

i 

; ranged  between  five  and  twenty-five  chances  of  success  per  ten 

\ thousand.        Needless  to  say,  even  the  slightest  variation  in  estimates, 

i 

i in  cases  such  as  this,  would  have  disproportionate  effects  due  to  the 

j small  order  of  magnitude  involved.  There  are  two  principle  methods 

!of  estimating  probability  of  success.   First  is  by  obtaining  the  sub- 

t 

ijective  judgements  of  persons  experienced  in  the  subject  area.  Second 

is  by  analysis  of  data  obtained  from  similar  prior  efforts.  The  main 
shortcoming  of  the  first  method  lies  in  the  bias  that  is  likely  to  be 
introduced.  The  second  is  found  disadvantageous  since  probability  of 
success  is  not  a  fixed  quantity,  but  is  a  function  of  the  state-of- 
the-art  and  level  of  knowledge  prevailing  at  the  time  of  the  estimate.-* 


*N.  R.  Baker  and  W.  H.  Pound,  "R  $  D  Project  Selection:  Where 
We  Stand,"  IEEE  Transactions  on  Engineering  Management,  EM- 11  (December, 
1964),  p.  129. 

2Asher,  "A  Linear  Programming  Model  for  the  Allocation  of  R  §  D 
Efforts,"  p.  155. 

3Rosen  and  Souder,  "A  Method  for  Allocating  R  5  D  Expenditures," 
p.  87. 
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.  A  final  complication  realized  in  using  probability  of  success 
estimates  is  that  there  is  no  reason  to  assume  that  the  probability  of 
success  and  level  of  funding  are  independent.  On  the  contrary,  the 
converse  is  almost  surely  true.  Despite  this,  only  the  method  of  Hess 
gives  consideration  to  the  apparent  dependency. 

Data  Requirements 

Since  the  descriptions  of  methods  given  earlier  were  somewhat 
abbreviated,  the  data  requirements  appear  deceptively  minor.   In  gen- 
eral all  techniques,  developed  to  date,  require  information  of  past 
efforts.  As  such,  many  of  the  methods  are  not  applicable  to  organiza- 
tions that  are  initiating  research  and  development  programs.   In  some 
cases,  such  as  that  of  Hess,  the  data  requirements  and  assumptions  are 
quite  restrictive.  As  Hess  points  out,  a  considerable  amount  of  work 
j would  be  required  before  his  method  could  be  applied. * 

The  fact  of  the  matter  is  that,  at  the  present  time,  there  is 
not  sufficient  data  available  upon  which  to  base  formally  structured 
research  and  development  selection  decisions.   Even  if  a  great  deal  of 
past  data  were  available,  it  is  not  clear  how  this  might  be  useful  in 
evaluating  current,  different  projects;  for  reasons  which  have  already 
been  discussed. 

As  a  result  of  this  paucity  of  "good"  information,  compensatory 
methods  have  been  undertaken.  As  was  mentioned  in  the  discussion  of 
probability,  one  approach  has  been  the  use  of  subjective  .estimates ,  an 


1Baker  and  Pound,  "R  S  D  Project  Selection:  Where  We  Stand, 
p.  128. 
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approach  requiring  judgmental  values  for  costs,  returns,  probabilities 
of  success,  and  so  forth.  The  difficulty  with  these  methods  has  been 
that  there  is  generally  a  prevalent  bias  which  may  render  the  data 
unreliable.   For  instance,  persons  wishing  to  "look  good"  will  con- 
sciously or  unconsciously  underestimate  the  potential  of  proposed 
projects.  On  the  other  hand,  there  are  the  incurable  optimists  who 
think  they  can  produce  results  worth  millions,  at  very  low  cost,  and 
in  a  short  period  of  time.   If  anything,  the  most  common  tendency 
is  towards  over  optimism.  This,  of  course,  is  the  error  that  is  most 
costly.3 

Optimization  Criteria 
As  the  term  "project  selection"  implies,  efforts  of  this  sort 
seek  to  evaluate  and  compare  alternative  proposals  for  the  purpose 
of  selecting  those  identified  as  holding  the  most  promise.   It  is  obvi- 
ous that  this  process  requires  a  basis  for  selection.  The  most  common 
criteria  that  are  used  are  some  form  of  profit  or  return  on  investment; 
either  can  be,  and  are,  computed  in  sundry  fashions.  This  limited  view 
of  what  constitutes  an  adequate  criterion  has  undoubtedly  grown  out  of 
the  tendency  of  business,  in  general,  to  use  profit  or  return  on  in- 
vestment. However,  as  Dean  and  Hauser  have  noted,  until  a  method  is 
formulated  for  application  in  a  particular  situation,  the  concern 


Ijbid. ,  p.  125. 

^R.  J.  Freeman,  "A  Stochastic  Model  for  Determining  the  Size 
and  Allocation  of  the  Research  Budget,"  p.  2.  . 

3R.  M.  Anderson,  "Handling  Risk  in  Defense  Contracting,"  Harvard 
Business  Review,  XLVII  (July-August,  1969),  p.  92. 
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should  not  be  with  the  development  of  criteria,  but  with  the  formula- 
tion of  models  which  are  adaptable  to  criteria.  The  more  general  in- 
tent should  be  to  generate  methods  which  enable  the  manager  to  syn- 
thesize the  information  that  is  available,  so  that  alternative  cri- 
teria can  be  examined. 

Progress  to  Date 
Several  studies  have  been  conducted  for  the  purpose  of  deter- 
mining the  extent  to  which  quantitative  techniques  have  been  incorpor- 

2 
j ated  into  program  selection  and  budgeting  efforts.   All  of  these 


have  indicated  that  use  of  these  techniques  has  been  slight;  despite 
the  current  trend  of  capitalizing  on  such  methods  in  other  manage- 
ment efforts. 

One  survey  concluded  that  a  major  reason  for  the  lack  of  signi- 
I  ficant  employment  has  been  that  the  methods  have  not  been  thoroughly 
I  tested,  using  real  data.   A  more  recent  examination,  completed  during 
the  year  prior  to  this  writing,  observed  that  a  more  likely  reason  is' 

that  the  rationale  for  the  selections  is  not  subject  to  external  vali- 

4 
dation  of  any  kind.   In  most  cases,  it  was  concluded,  the  only  test  of 


Dean  and  Hauser,  "Advanced  Materiel  Systems  Planning,"  p.  22. 

Baker  and  Pound,  "R  and  D  Project  Selection:  Where  We  Stand," 
and  Cetron,  Martino  and  Reopke,  "The  Selection  of  R  5  D  Program  Content- 
-Survey  of  Quantitative  Methods." 

3Baker  and  Pound,  "R  and  D  Project  Selection:  Where  We  Stand," 
p.  130.  | 

4 
Robert  Ayres,  Technological  Forecasting  and  Long-Range  Plan-  - 

ning  (New  York:  McGraw-Hill  Book  Company,  1969),  p.  198. 
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validity  is  the  internal  consistency  and  reasonableness  of  the  com- 
parison procedures.   Such  a  criterion,  however,  is  usually  only  fully 
satisfactory  to  the  model  designer. 

One  should  not  be  too  hasty,  though,  in  discounting  the 
utility  of  quantitative  techniques  simply  because  "they  do  not  prove 
out".  This,  frequently,  is  a  characteristic  of  any  statistical  de- 
cision-making technique. 

Two  conclusions  might  be  made  at  this  point.   The  first  is 
I  that  the  problem  at  hand,  that  of  project  selection,  and  so  on,  may  be 
deserving  of  some  attention  to  developing  methods  of  solution  which  are 
more  practical  in  nature,  vice  those  which  are  theoretically  more 
precise.   Second  is  a  need  to  recognize  that  quantitative  techniques 
should  not  be  expected  to  replace  the  decision-maker.  The  impact  of 
these  techniques  is  not  in  problem  solving,  but  in  problem  formulation: 
the  way  in  which  managers  conceptualize  their  problems. 


CHAPTER  II 

DESCRIPTION  OF  THE  TECHNIQUE 

The  techniques  to  be  discussed  in  this  chapter  are  quantita- 
tive techniques  similar  to  those  examined  in  the  preceding  chapter. 
|  Before  continuing,  though,  it  should  be  recognized  that  techniques 
|  such  as  these  are  not  used  to  any  great  extent  in  existing  research 
land  development  management  systems.  Most  systems,  today,  still  rely 
on  the  basic  methods  of  more  conventional  systems;  being  oriented 
towards  control  on  the  basis  of  budgetary  variances. 

The  management  of  research  and  development  solely  on  the 
basis  of  financial  considerations,  without  a  sincere  attempt  to  in- 
corporate progress  reporting,  in  some  instances  degenerates  into  a 
contest  among  program  managers,  which  tests  political  rather  than' 
technical  skills.   Several  rather  familiar  methods  of  program  manage- 
ment have  been  spawned  under  these  conditions.  One  of  these  is  the 
squeaking  wheel   method  of  allocating  resources.  Using  this  method,  a 
manager,  preparing  for  the  allocation  of  funds,  will  reduce  resources 
for  all  projects  and  then  wait  to  see  which  project,  manager  expresses 
the  loudest  complaint.  On  the  basis  of  the  most  insistent  squeaking 
then,  resources  will  be  restored  until  budgetary  levels  are  reached. 
A  second  popular  management  technique  is  that  which  seeks  to 
perpetuate  the  Glorious  Past.     This  method  dictates  that  a  group  within 
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the  organization,  which  has  recently  demonstrated  notable  success, 
should  gain  acceptance  of  research  proposals  for  the  next  year,  or 

five  years,  or  for  some  other  appropriate  length  of  time.  This  doc- 

I 
trine,  which  is  founded  on  the  premise  of  "once  successful,  always 

successful"  permits  the  pursuit  of  research,  independent  of  usefulness 

or  efficiency. 

A  final  management  approach  to  resource  allocation  is  based  on 

the  white  charger   technique.  To  implement  this  method,  management  calls 

on  various  departments  or  groups  within  the  organization  to  hold  forth 

with  well-rehearsed  presentations  complete  with  multi-color  graphs  and 

handouts.  That  group  which  provides  the  most  impressive  forensic  dis- 

i  ! 

play  is  then  rewarded  with  increased  resources. 

Though  techniques  such  as  these  seem  to  be  tragi-comic  in  the 
context  of  management  science,  their  popularity  and  frequency  of  use 
is  substantial.   The  fact  that  methods  such  as  these  are  used  to  the 
extent  indicated  serves  to  illustrate  the  need  for  objective  consider- 
ation of  progress  and  achievement  in  research  and  development. 

! 

The  process  of  program  selection,  regardless  of  the  type  of 
quantitative  selection  technique  that  is  employed,  must  include  cer- 
tain operations  and  procedures.  These  necessary  steps,  common  to  all 
selection  processes,  constitute  the  subject  material  of  this  chapter. 
After  these  steps  are  examined  and  analyzed,  the  following  chapter  will 
describe  the  methods  to  be  utilized. 


M.  J.  Cetron,  "Technological  Forecasting:   A  Prescription  for" 
the  Military  R5D  Manager,"  Naval  War  College  Review,  XXI  (April,  1969) 
p.  29. 
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Project  Selection 
The  process  of  selecting  research  and  development  projects  is 
closely  akin  to  most  other  decision-making  processes  in  that  it  re- 
quires the  evaluation  of  alternatives  preparatory  to  actual  selection. 
In  this  respect,  then,  project  evaluation  implies  a  priori   technical 
evaluation  to  determine  the  degree  to  which  projects  contribute  to  the 
accomplishment  of  established  goals  and  objectives.  This  is  in  con- 
j trast  to  a  conventional  meaning  of  evaluation,  which  connotes  a  post- 
mortem to  analyze  the  effectiveness  and  efficiency  of  the  conduct  of 
an  accomplished  project.   Project  evaluation  is,  therefore,  intended  to 
originate  some  sort  of  a  characteristic  quantity  or  expression  which 
represents  a  project  and  will  enable  a  value  comparison  with  other 

i 

I  projects  competing  for  funds  and  resources. 

Objectives 

Prior  to  the  commencement  of  project  evaluation,  it  is 
necessary  to  identify  some  broad  objectives  of  the  entire  selection 
process.  These  objectives  may  simply  be  in  a  statement  of  guiding 
policies,  or  they  may  be  more  closely  defined. 

Research  and  development  objectives  are  usually  products  of 
long-range  planning.  This  planning  process  commences,  as  does  plan- 
ning for  non-research  and  development  activities,  with  consideration 
of  existing  plans  and  policies,  the  state-of-the-art  in  disciplines 
of  interest  and  the  competitive  environment.  The  results  of  this 
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portion  of  the  planning  process  can  be  considered  to  be  "strategic 
objectives"  since  they  derive  from  consideration  of  current  re- 
search and  development  conditions  and  focus  on  projected  conditions 
and  competitive  position. 

The  nature  of  strategic  objectives  can  be  illustrated  if  one 

2 
considers  a  hypothetical  case:  Cunisibe  Metals,  Inc.   is  a  medium-size 

materials  research  organization  which  limits  its  efforts  to  non-ferrous 
[ applications.   In  the  past,  Cunisibe  has  usually  expended  about  75  per 
'cent  of  its  basic  research  budget  in  the  copper-bronze  field.  However, 
\ Cunisibe" s  position  in  this  field  has  never  improved  substantially  in 
j the  face  of  considerable  competition  from  the  research  divisions  of 
ithe  nation's  three  largest  copper-producing  companies. 

Cunisibe' s  most  recent  planning  effort  has  high-lighted  two 

particular  environmental  factors: 

1)  The  questionable  long-range  position  of  copper-bronze 
materials  due  to  recent  shortages  of  domestic  raw  materials  and  an 
even  more  questionable  stability  among  African  suppliers. 

2)  A  growing  interest  in  ocean-engineering  and  efforts  to 
exploit  ocean-resources. 

As  a  result  of  these  factors,  then,  Cunisibe  formulated  a 
strategic  objective  aimed  at  gaining  a  national  position  based  on  an 


The  terminology  used  in  this  section  regards  strategic,  man- 
agement and  operational  objectives,  will  correlate  with  the  terminology 
of  R.  N.  Anthony  in  Planning  and  Control  Systems  --  A  Framework  for 
Analysis  (Boston:   Harvard  University,  1965). 

This  is  a  fictitious  organization.  However,  the  circumstances 
described  resemble  those  of  an  actual  research  organization. 


27 


expertise  in  developing  materials  appropriate  for  use  in  the  ocean 
environment.  This  strategic  objective  will  cause  Cunisibe  to  shift 
emphasis  from  copper-bronze  research  to  endeavors  in  utilizing  such 
materials  as  tungsten  and  magesium.   Likewise,  it  will  open  new  dis- 
ciplines, such  as  galvanic-corrosion  control. 

Strategic  objectives,  however,  cannot  afford  adequate  guidance 
for  immediate  applications.  They  simply  identify  long-term  needs. 
The  next  step,  then,  towards  establishing  operational  requirements  is 
to  determine  the  nature  of  corporate  deficiencies  regards  the  es- 
tablished strategic  objectives.   This  portion  of  the  planning  process 
is  essentially  a  deficiency  analysis  of  current  capabilities  and  re- 
sources. The  result  of  this  analysis  will  be  "management  objectives" 
which  will  be  applicable  in  short-term  planning.  These  management 
objectives  will  identify  subject  areas  deserving  attention  in  organ- 
izing programs.   Following  the  previous  example,  management  objectives 
at  Cunisibe  for  the  first  few  years  might  include: 

1)  Acquisition  of  personnel  with  expertise  in  the  fields  of 
Tungsten  alloying  processes,  scintered  metallurgy  and  metal-fibre 
growth  techniques. 

2)  Development  of  research  facilities  adequate  to  support 
new  areas  of  interest. 

An  examination  of  management  objectives  would  reveal  that, 
though  they  are  more  precise  than  stategic  objectives,  they  are  too 
general  to  be  immediately  useful.  Thus,  it  is  necessary  to  translate 
management  objectives  into  "operational  objectives"  which  can  be 


28 


directly  related  to  particular  research  endeavors.  These  operational 
objectives  are  the  criteria  that  are  used  in  the  comparative  analysis 
of  research  proposals. 

Operational  objectives  are  usually  couched  in  terms  of  certain 
technologies  or  topic  areas  which  have  been  identified  as  being  of 
interest.  However,  for  production-oriented,  commercial  firms  oper- 
ational objectives  may  be  related  in  more  economic  terms.  This,  nat- 
jurally,  results  from  the  fact  that  such  firms  have  objectives  which, 
! generally,  are  more  readily  expressed  in  terms  of  economic  indicators. 


For  example,  one  market-oriented  firm  adopted  the  following  as  the 
objectives  of  research  and  development  programs: 

1)  To  maximize  expected  profit. 

2)  To  maximize  expected  research  successes. 

3)  To  achieve  a  return  on  expenditure  (ratio  of  maximum  ex- 
pected profit  to  total  expended)  of  at  least  55.0. 

Regardless  of  whether  a  research  and  development  organization 
is  profit  oriented  or  non-profit,  or  devoted  to  product  development  or 
process  research,  the  exercise  of  formulating  objectives  is  essential 
to  any  attempts  at  selecting  program  content  on  the  basis  of  struc- 
tured, quantitative  procedures.  As  will  be  described  in  following 
sections,  these  objectives  are  needed  if  candidate  programs  are  to 
be  compared  in  an  objective  fashion,  and  if  ultimate  program  progress 
is  to  be  monitored  meaningfully. 


p.  90, 


Rosen  and  Souder,  "A  Method  of  Allocating  R  S  D  Expenditures," 
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Utility  Analysis 

A  significant  characteristic  of  the  process  of  formulating 
objectives,  that  has  been  described  in  the  preceding,  is  that  it  is 
almost  exclusively  a  qualitative  exercise.  Consequently,  the  de- 
cisions, made  in  arriving  at  objectives,  have  little  utility  in  terms 
of  a  quantitative  selection  technique.  This  section  will  be  devoted 
to  describing  some  traditional  operations  research  procedures  which 
can  be  used  to  relate  quantitative  techniques  to  qualitative  ob- 
jectives. 

In  the  preceding  example  the  formulation  of  objectives  progres- 
sed to  the  point  of  operational  objectives,  which  are  to  provide  the 

necessary  decision  rules  for  program  selection.  The  next  step  in  the 

j 
analysis  must  then  cast  program  alternatives  in  terms  which  are 

amenable  to  manipulation,  in  order  that  decision  rules  may  be  applied. 

The  quantitative  characterization  of  alternatives  is  most 

j  easily  done  using  "figures  of  merit"  or  "value  functions".   Either 

of  these  terms  refers  to  the  assignment  of  a  number  or  expression,  to 

each  alternative,  which  represents  the  utility  or  desireability  of 

the  alternative.  The  measurement  of  utility  in  this  fashion  is  an 

essential  ingredient  of  traditional  operations  research  methods  such 

as  game  theory,  allocation  problems  and  solutions  of  competitive 

situations.   Though  a  variety  of  utility  measurement  methods  are 

available,  an  examination  of  two  of  them  will  adequately  illustrate 


*R.  L.  Ackoff  and  M.  W.  Sasieni,  Fundamentals  of  Operations 
Research  (New  York:  John  Wiley  and  Sons,  1968),  pp.  49-56. 
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the  principles  involved. 

Utility  analysis,  seeks  to  determine  the  value  or  value  function 
to  be  attached  to  each  research  proposal.  Value,  here,  should  be 
thought  of  as  expressing  the  "net  worth"  of  a  proposal  to  the  re- 
search organization.   It  has  been  mentioned  previously,  and  deserves 
re-emphasis,  that  the  measure  of  value  is  a  function  of  objectives. 
Thus,  the  rules  for  value  assessment  must  be  known  before  assignment 

i  is  made. 

To  illustrate  the  nature  of  the  value  assessment  problem,  con- 

I  sider  the  case  of  storage  battery  technology,  and  the  problem  of 

, assessing  the  importance  of  three  battery  parameters:  volume,  cost 

I  I 

! and  time  between  recharging.  Assume  that  the  importance  of  each 

! parameter  will  be  indicated  by  assigning  to  each  a  number  ranging 
j 

I  from  10,  for  great  importance,  to  0  for  no  importance.   Further, 

I 

j assume  that  an  assessment  of  importance,  or  value,  will  be  made  by 

each  of  the  following: 

1)  the  user;  a  Lieutenant,  U.S.  Navy,  commanding  a  boat  con- 
taining batteries  and  drifting  on  a  Vietnamese  river  on  night  patrol. 

2)  the  R  5  D  manager;  an  Admiral,  responsible  for  the  Navy's 
total  research  and  development  program,  while  considering  next  year's 
budget. 

3)  the  boat  designer;  a  naval  architect  designing  a  boat  for 
use  in  Vietnam. 


1M.  J.  Cetron,  "Technological  Forecasting:  A  Prescription  for-j 
the  Military  R5D  Manager,"  p.  29. 
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4)  the  R  5  D  engineer;  a  project  engineer  working  in  a  Navy 
laboratory  and  seeking  to  improve  the  general  performance  of  batteries 

Obviously,  each  of  these  individuals  will  consider  battery 
parameters  from  different  points  of  view.   It  should  not  be  surprising 
if  the  responses  of  the  four  individuals  were  as  different  as  those 
shown  in  Table  1. 

TABLE  1 

BATTERY  PARAMETER  RATINGS 

Parameter     User    Manager  Designer    Engineer 

Volume        3       2  10         8 

Cost         0       10  2         2 

Time  Between     10       2  4         1 

Recharging 

If  the  operational  objective  of  a  batter)'  development  program 
were  to  provide  the  best  battery  for  combat  units,  the  value  assess- 
ment of  the  user  would  likely  be  the  most  accurate.  However,  if  the 
objective  were  to  maintain  some  battery  research  and  development  de- 

i spite  budget  cuts,  the  manager's  assessment  would  be  appropriate. 

j 

i  Similarly,  other  objectives  would  lead  to  the  selection  of  either  the 

; designer's  or  the  engineer's  assessment. 

This  example  should  serve  to  illustrate  the  important  fact 

:  that  research  and  development  programs  have  no  intrinsic  value  or 

I 

|  worth.  Value  must  be  measured  with  consideration  given  to  objectives. 
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The  selection  problem  involving  alternatives  having  n  signif- 
icant parameters  takes  the  general  form  of 


n        n 

vk  =  I     wki  =  I     aixki  ai-1} 

i=l       i=l 


where   v^  =  the  value  of  the  kth  alternative 


Wj,j,  =  the  value  of  the  ith  parameter  of  the  kth  alternative 

a^  =  the  weight  of  the  ith  parameter 

xki  =  t*ie  i-tn  parameter  of  the  kth  alternative. 


The  preceding  example  also  shows  a  method  of  translating 

| 
: qualitative  judgements  into  quantitative  measures.   Before  conducting 

!  ! 

the  exercise  associated  with  the  data  in  Table  1,  the  R  §  D  engineer 

; could  have  related  that  he  considered  battery  volume  to  be  more 

significant  than  cost  or  time  between  recharging.   This  observation, 

i 
however,  would  be  of  limited  use  to  someone  selecting  a  battery  on 

the  basis  of  the  engineer's  criteria.   After  the  process  of  rating 
the  parameters  on  a  0  to  10  scale,  though,  one  realizes  that  the 
engineer  regards  volume  as  being  four  times  as  important  as  cost  and 
eight  times  as  important  as  time  between  recharging.  Now,  a  quantita- 
tive selection  is  possible. 

Selection 


^Freeman,  "A  Stochastic  Model  for  Determining  the  Size  and 
Allocation  of  the  Research  Budget,"  p.  2. 


L_ 


33 


-  This  type  of  selection  can  be  demonstrated  by  considering 
three  hypothetical  alternative  batteries;  the  parameters  of  which  are 
shown  in  Table  2.  Applying  equation  (II-l)  to  the  data  in  Table  2  (the 


TABLE  2 

BATTERY 

PARAMETERS 

Parameter 

n 

#2     #3 

Volume  (ft3) 

.80 

1.10    .95 

Cost  ($) 

110 

80     87 

Time  Between 

16 

12     14 

recharging  (hr) 

data  being  normalized  by  the  mean  for  each  parameter),  and  using  the 
weighting  values  given  by  the  engineer  in  Table  2,  one  obtains 

,r        -     R  v  -80  *  9  v  110  *  1   v  16 

=  10.26 
v2  =  12.80 
v3  =  10.90  . 

Since  it  is  presumed  that  one  would  seek  minimum  cost,  volume 
and  time  between  recharging,  the  alternative  having  minimum  value  (v) 
would  be  selected;  that  is,  battery  number  1.   It  should  be  noted  that 
the  results  of  the  selection  process  would  have  been  substantially 
different  had  other  criteria  from  Table  1  been  employed. 

The  preceding  discussion  illustrates  the  essential  character- 
istics of  the  selection  problem.  First,  the  problem  involves  numerous 
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alternatives;  say  n.   Each  of  these  has  a  present  net  value,  V.. 
Though  not  mentioned  explicitly,  before,  each  alternative  will  also 
have  an  associated  cost,  C.  .   If  one  assumes  that  alternatives  are 
independent,  and  that  the  V.  and  C.  are  sufficient  estimates  of  actual 
values,  then  the  general  selection  problem  is  to  simply  find 

n 
max  I     VL  (II-2) 

i=l 

where  VL   =  f^Ci)  .  (II-3) 

Unconstrained,  the  solution  of  this  problem  is  trivial  since  it  amounts 
to  simply  selecting  all  alternatives. 1 

A  constraint  that  is  commonly  encountered  in  selection  efforts, 
is  that  which  limits  available  funds,  in  the  form  of  a  budget.   In 
ithis  case »  for  T  years  in  the  planning  period 


q  =  I    cit  Cii-4) 

t=i 


where  c^t  is  the  cost  during  year  t.   If  an  annual  budget,  Bt,  must 
be  observed,  then  the  solution  is  subject  to 


n 

I     uicit  1  Bt  >     t  =  1,  ...,  T  .  (II-5) 

i=l 


Engineering  Economy  Division,  American  Society  for  Engineering 
Education.   The  Fourth  Summer  Symposium  Papers  (Hoboken,-  New  Jersey, 
1966),  p.  65: 
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where .  u-  =  1  if  alternative  i  is  selected, 
u^  =  0  if  alternative  i  is  rejected, 

Interdependence 


The  general  selection  problem,  described  by  equations  (II-2) 
through  (II-5),  can  be  expanded  by  admitting  interdependence  among 
alternatives.  This  provision  would  result  in 

Vi  =  f^q,  ....  q,  ....  Cn)  ;  (II-6) 

that  is,  a  given  alternative  may  be  affected  by  expenditures  for  other 

alternatives.  The  most  common  example  of  this  situation  arises  when 

two  or  more  different  research  projects  are  being  conducted  by  the 

same  working  group;  for  example  an  organic  chemistry  branch.   In 

situations  such  as  these,  though  the  research  projects  are  different 

in  substance  and  structure,  by  virtue  of  the  mere  fact  they  all  relate 

to  organic  chemistry,  it  would  be  expected  that  some  interdependence 

would  exist. 

As  soon  as  the  possibility  of  interdependence  is  admitted, 

the  nature  of  the  selection  problem  changes  substantially.  Though 

the  objective  of  maximizing  expected  value,  as  expressed  in  equation 

i 
(II-2)  is  still  applicable,  it  is  no  longer  a  simple,  or  trivial 

problem.   Recognizing  the  implications  of  equation  (II-6),  which  can 

be  rewritten  as 


*  C  , 
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where  b_  is  the  transition  matrix,  relating  £  to  V,  it  becomes  apparent 
that  some  techniques,  other  than  simple  rank-ordering,  will  be  re- 
quired to  obtain  a  solution  to  this  selection  problem. 

The  first  step  in  treating  the  consideration  of  interdepen- 
dence is  to  estimate  its  quantitative  characteristics.  The  first 
major  attempt  to  do  just  this  was  made  in  Project  PATTERN.   This 
project  sought  to  examine  the  contribution  of  so-called  "cross-support" 
among  research  projects,  and  to  use  the  information,  thus  gained,  in 
planning  future  research  efforts. 

In  the  PATTERN  report,  an  example  research  program,  a 
j  Strategic  Manned  Base  Orbital  System,  was  compared  with  three  other 
J  related  projects.  The  objective  of  this  comparison  was  to  determine 
the  progress  to  be  realized  in  the  Orbital  System  research  if  success 
were  achieved  in  any  one  of  the  other  three  projects.  The  results  of 
the  cross-support  analysis  are  shown  in  Table  3. 

The  data  in  Table  3  relate  to  certain  elements  of  the  Orbital 
System  program  (for  instance,  elements  4,  32,  33,  106,  and  so  forth). 
For  each  of  these  elements,  the  degree  of  correlation  with  the  re- 
search objectives  of  the  three  other  projects  being  examined  are 
shown.   For  example,  element  number  123  of  the  Orbital  System  project 
had  a  .05  correlation  with  elements  1531  and  1532  of  the  Infra-red  and 
Radar  Ranging  project.  That  is,  if  the  objectives  of  the  Infra-red 


Aaron  L.  Jestice,  Project  Pattern,  Presented  to  the  Joint  Na- 
tional Meeting,  Operations  Research  Society  of  America,  October  7,  1969 
(Minneapolis,  Minn.:  The  Institute  of  Management  Sciences,  1964),  p. 15. 
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TABLE  3 

CROSS  SUPPORT  ANALYSIS  OF 
STRATEGIC  MANNED  BASE  ORBITAL  SYSTEM 

Manned 
Base 
Orbital 
SyqtPTTi 

Hi  Scan  Rate 

IR  and  Radar 

Ranging 

Hi  Scan  Rate 
Laser  Range 

and  Angle 
Radar  : 

Hi  Scan  Rate 

Radar  (10-30  KMC) 

(1  MW  Peak) 

Project 

Project 

Cross 

Support 

Project 

Cross 

Support 

Project 

Cross 

Support 

4 

32 

1752 

0.10 

33 

1753 

0.40 

106 

121 

1504 

0.50 

123 

1551 
1532 

0.05 
0.05 

138 

286 

1524 

0.30 

316 

1520 
1525 

0.20 
0.20 

317 

1529 
1530 

0.05 
0.05 

i 

586 

1 

589 

633 

1760 

0.80 

1759 

0.08 

636 

1762 

0.70 

176. 

0.05 

637 

1764 

0.90 

1763 

0.06 

638 

1766 

0.70 

1765 

0.05 

649 

1768 

0.60 

1767 

0.03 

Source:   Jestice,  Project  Pattern,  p.  15. 
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project  are  achieved,  then  .05  of  the  effort  to  achieve  Orbital 
System  objectives  would  be  unnecessary. 

The  significance  of  cross-support,  or  interdependence,  among 

research  projects  is  readily  demonstrated.   For  example,  assume  that 

1 
the  Orbital  System  project  has  been  selected  for  funding.  Now,  con- 

! 
sider  the  problem  of  selecting  one  additional  program,  from  among 

J 
the  three  alternatives  shown  in  Figure  3.   If  it  is  further  assumed 

that  the  objectives  of  the  three  alternatives  are  of  equal  importance, 
and  that  the  independent  costs  are  the  same,  then,  obviously,  one 
would  select  the  Laser  Range  and  Angle  Radar  project  for  funding. 
This  is  so  because  the  actual  costs  will  be  substantially  less  than 
the  independent  costs  due  to  the  degree  of  interdependence  indicated 
by  the  data  of  Table  3. 

It  should  be  noted  that  the  measurement  of  cross-support,  as 
developed  for  Pattern,  is  a  technique  for  quantifying  the  interde- 
pendence of  research  projects  by  examining  the  direct  relationships 
among  the  projects.  Another  approach  to  the  problem  of  accounting 
for  interdependence  is  to  measure  project  relationship  to  some  ob- 
jectives other  than  the  interdependent  projects.  A  typical  applica- 
tion of  this  type  of  analysis  is  to  examine  the  contributions  of  a 
given  project  to  a  number  of  selected  technological  disciplines. 

One  method  for  measuring  project  content  in  this  manner  is 

! 
that  of  PROFILE  (Programmed  Functional  Indices  for  Laboratory  Eval- 

1 

uation).   This  procedure,  developed  for  evaluation  of  military 


1Erick  Jantsch,  Technological  Forecasting  In  Perspective  (Paris: 
Organization  for  Economic  Co-operation  and  Development,  1967),  p.  228. 
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research  projects,  relates  project  content  to  various  military  missions 

The  type  of  analysis  utilized  in  PROFILE  is  an  analysis  of  program 

relevence  to  prescribed  objectives. 

Relevence  evaluation  is  most  easily  described  by  constructing 

a  matrix  to  display  the  form  of  the  analysis.   As  shown  in  Table  4, 

each  of  the  n  program  alternatives  is  related  to  the  objectives,  a 

through  v,  by  the  significance  numbers,  s.   In  addition,  each  of  the 

objectives  is  assigned  a  weight,  q,  which  indicates  its  importance 

I  relative  to  the  other  objectives, 
i 

The  desireability  of  research  programs  can  then  be  measured 


by  relevence  numbers,  r,  which  are  obtained  from 


X=cc 


To  ensure  the  homogeneity  of  the  selection  logic,  the  data  in  Table  4 
is  normalized  subject  to 


I     qx  «  1  (H-9) 

X=a 


1Ibid. ,  p.  221. 
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TABLE  4  ' 
RELEVENCE  EVALUATION  MATRIX 


Objectives 

Weights 

of 

Objectives 

Program  Alternatives 

a 

b 

c 

... 

j 

... 

n 

a 

qa 

sa 
b  a 

A 

s  c 

*aj 

sa 
5  n 

8 

% 

S2 
s  a 

A 

SB 

s  c 

^ 

S6 

s  n 

Y 

qY 

sYa 

*Tb 

*Yc 

s7j 

*Yn 

• 

^X 

^X 

sXa 

sxb 

=  Xc 

sX: 

• 

sXn 

'. 

V 

<lv 

=Va 

svb 

*Vc 

sVJ 

*Vn 

rai 

''' 

rci 

... 

^ 

... 

T\ 

Source:  Jantsch,  Technological  Forecasting  in  Perspective 
p.  221. 
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and 


n 

I     s    .  =  1  .  (11-10) 

j=a 


Summary  of  Evaluation  Techniques 


The  process  of  selecting  research  and  development  programs,  as 
it  has  been  described,  is  one  of  sequential  decision-making.   However, 
too  frequently,  the  sequential  aspects  of  such  an  endeavor  are  not 
recognized  by  the  research  and  development  manager.  These  sequential 
characteristics  are  of  prime  importance  in  the  description  and  struc- 
turing of  the  objectives  and  criteria  to  be  used  in  project  selection. 
The  primary  reason  for  selecting  particular  projects  from  those  avail- 
able is  to  maximize  the  ultimate  benefit  to  the  organization  and  to 
maintain  some  assurance  that  research  efforts  will  be  productive.  As 
long  as  objectives,  and  the  relations  among  them,  remain  unclear,  it 
becomes  doubtful  that  any  selection  method  will  achieve  much  success. 
A  common  failing  of  selection  techniques  lies  in  a  disregard  for 
proper  planning  and  analysis  in  the  formulation  of  selection  systems. 

The  preceding  description  of  project  evaluation  and  selection 
principles  has  been  presented  in  terms  of  the  sequential  decision- 
making process.   Figure  2  displays  the  activities  involved  and  the 
manner  in  which  they  are  related.  The  description  of  the  flow  of 
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FIGURE  2 
EVALUATION  AND  SELECTION  PROCESS 
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FIGURE  2--Continued 
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activity  is  intended  to  emphasize  the  serial  nature  of  the  process. 
It  should  be  emphasized  that,  once  operational  objectives  are  estab- 
lished, no  further  feedback  function  is  encountered  until  the  eval- 
uation/selection exercise.  The  significance  of  this  lies  in  the 
understanding  that  modifying  objectives   for  the  purpose  of  making 
particular  programs  acceptable,  defeats  the  intent  of  an  organized 
selection  effort. 

Implementation 

i 

The  selection  of  research  and  development  projects  is  essen- 
tially a  planning  function.   It  commences  with  long-range  plans  and 
|  forecasts,  and  results  in  short-term  plans  for  research  efforts.  The 
i  next  step,  then,  in  funding  of  research  and  development,  is  the 
!  process  of  implementing  the  selection  decisions. 

i  i 

Time  and  Cost  Parameters 

The  two  most  obvious  characteristics  of  a  research  project, 
other  than  its  subject  matter,  are  the  cost  of  the  project  and  the 
time  until  completion.  The  importance  of  cost  is  obvious,  for  the 
undertaking  of  an  organized  selection  effort  is  initiated  by  a  need  to 
allocate  limited,  financial  resources  among  numerous  candidate  re- 
search efforts.   Cost  has  added  significance  for  those  programs  which 

i 

span  more  than  one  budgetary  period  (usually  a  year).  This  results 
from  the  fact  that  monetary  contraints  are  usually  based  on  the  dollar 

I 

*Souder,  "Experiences  with  an  R  5  D  Project  Control  Model, "• 
p.  40. 
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requirements  of  research  programs  for  a  given  period,  rather  than  the 
dollar  requirements  over  the  life  of  the  project.   In  this  regard,  then 
the  research  and  development  manager  is  usually  interested  in  the  dis- 
tribution of  funds  over  the  life  of  the  project. 

Project  cost  can  be  measured  in  numerous  ways,  and  the  method 
to  be  used  is  strongly  a  function  of  the  nature  of  the  organization 
conducting  the  research  and  development.   In  most  commercial  enter- 
prises, for  instance,  a  project  incurs  not  only  direct  costs  for 
material  and  labor,  but  also  period  costs  and  opportunity  costs  as 
indicated  in  Figure  3. 

One  example,  however,  of  research  and  development  funding 
which  is  not  constrained  by  annual  budgetary  limitations,  is  govern- 
Iment  programs,  which  are  the  subject  of  this  paper.  Government  re- 
search programs  are  generally  selected  on  the  basis  of  total  cost  with 
allowance  made  for  the  fact  that  appropriate  annual  fund  appropriations 
will  be  forthcoming. 

Another  aspect,  of  cost  distribution,  other  than  that  con- 
cerning period  budget  constraints,  addresses  the  question  of 
efficiency  in  the  rate  of  expenditure.  At  the  root  of  this  consider- 
ation is  the  fact  that  research  achievement  is  not  a  linear  function 
of  funds  expended.  At  times,  research  programs  are  funded  on  a  "crash" 
jbasis,  with  available  monies  being  increased  considerably,  in  antici- 
pation of  realizing  earlier  success.  Though  project  completion  may  be 


Rosenbloom,  "Notes  on  the  Development  of  Network  Models  for 
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FIGURE  3 
PROJECT  COST  FUNCTIONS 
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_______  j 

expedited  in  such  a  fashion,  there  exists  an  inherent  wastage  in  this 

type  of  effort.  This  wastage  will  result  primarily  from  less 

i 
effective  control  and  co-ordination  and  less  efficient  utilization  of 


learning  processes.   Thus,  wastage  will  occur  when  expenditures  exceed 
some  "satiation"  level,  as  discussed  in  Chapter  I. 

One  study,  of  the  expenditure-achievement  mechanism,  found 

that  achievement,  as  a  function  of  expenditure,  approximates  an  ex- 

1 

| ponential  function.   This  study  hypothesized  that  there  exists,  for 

I  the  ith  period  of  a  program,  some  expenditure,  y.  ,  which  can  be  con- 
I  sidered  efficient.  Then,  if  x^  is  the  actual  expenditure  for  that 
;  period,  the  effective  expenditure,  zj,  or  that  expenditure  which 
I  actually  contributes  to  achievement,  can  be  found  from 


H  =  y_  (y~)  *   ;  xi  >  yi,  £i  <  1 
zi  =  xi  ;  xi  <  xi 

where  e^  is  a  characteristic  of  the  research  project. 

A  final  aspect  of  the  study  concluded  that  wastage,  adherring 
to  inflated  expenditure  rates,  was  more  pronounced  during  the  initial 
stages  of  a  research  program,  and  decreased  in  significance  as  the 
effort  matured.   Initial  activity  in  research  programs  is  generally 
devoted  to  seeking  direction  and  definition.  Consequently,  added 


1 Atkinson  and  Bobis,  "A  Mathematical  Basis  for  the  Selection 
of  Research  Projects,"  p.  5. 
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funds  made  available  during  this  period  will  most  likely  be  applied  to 
parallel  exploration.   The  results  of  these  funds,  ultimately,  will 
not  be  germane  to  the  research  effort  and  therefore,  must  be  con- 
sidered wasted  for  purposes  of  the  project.   However,  the  acceleration 

i 
of  funding  in  the  latter  stages  of  a  project  will  result  in  parallel 

endeavors  in  more  closely  defined  areas,  such  as  development,  and  will 
be  more  productive  in  terms  of  useful  results. 

Another  study  of  considerable  interest,  in  this  regard,  was 
that  conducted  by  Hess,  as  described  in  Chapter  I.  As  a  result  of 
this  study,  which  examined  the  optimal  expenditure  for  a  given  period 
j of  a  project,  and  gave  consideration  to  the  history  of  preceding  ex- 
penditures, Hess  concluded  that,  in  general,  actual  expenditure 
practices  closely  approximated  the  theoretical  optimum.   One  can  con- 
clude from  this,  that  the  prescriptive,  or  intuitive  practices, 
which  are  still  the  foundation  for  most  research  and  development 
funding  procedures,  are  substantially  adequate.   Indeed,  one  study, 
which  examined  precisely  the  question  of  subjective  forecasts  and 
estimates  by  research  and  development  managers,  found  that  prescriptive 
practices  are  not  unreliable,  and  the  concomitant  subjective  forecasts 
can  be  extremely  accurate. 

From  the  foregoing,  then,  one  must  recognize  that  expenditure 
rates  in  research  programs,  that  is  the  distribution  of  expenditures 
over  the  life  of  the  program,  is  worthy  of  consideration.  However, 


Hess,  "A  Dynamic  Programming  Approach  to  R  and  D  Budgeting 
and  Project  Selection,"  p.  178. 
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analytic  techniques  which  address  this  problem  are  ultimately  founded 

i 
on  subjective  estimates  from  operating  personnel.   Further,  there  is 

every  indication  that  subjective  estimates  of  expenditure  rate  require- 
ments are  quite  adequate. 

The  second  significant  parameter  of  a  research  project,  that 
of  time  until  completion,  has  two  important  areas  of  impact  in  the 
value  of  a  project.   First  is  the  added  direct  cost  due  to  delay.  That 
lis,  as  continuation  of  a  project  is  experienced,  the  cost  of  maintain- 
ing the  research  team  and  providing  services  is  encountered.   Second, 
land  probably  more  important,  is  the  decreased  utility  of  the  results 
I  of  the  research  due  to  the  delay.   In  commercial  enterprises  this  may 

|be  realized  in  diminished,  competitive  position.   In  military  re- 

I 

j search  and  development  programs,  the  results  may  be  of  even  more 

! significance,  such  as  the  diminished  strategic  position  which  has 

resulted  from  the  delayed  development  of  the  F- 111  fighter-bomber. 

The  immediate  consequences  of  extended  time  in  research, 
though,  are  usually  subordinate  to  those  resulting  from  extensions  in 
expenditures.  Most  organizations  operate  within  well  defined  limita- 
tions on  financial  resources.  On  the  contrary,  time  available  is  of 
infinite  supply. 

Quantitative  techniques  devoted  to  the  consideration  of  the 
time  factor  in  research  are  usually  limited  to  estimates  of  expected 
value;  intrinsic  or  commercial.  One  study,  for  example,  estimated 
that  the  commercial  value  of  the  product  of  research  would  decrease 
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by 

some  factor,  k, 

which 

is  derived 

from 

pro] 

ected  sales  patterns. 

Therefore,  if  V  is 

0 

the 

original  estimate  of 

commercial 

value,  based 

on 

an  assumed  compl 

etion 

date,  the  value 

after  n  years 

of  delay  would 

be 

vn  =  v0  (1  - 

-  k)n 

• 

|  Summary  of  Implementation  Procedures 

Time  and  cost  in  research  programs  are  quantities  that  are 
|  easily  measured  and  readily  incorporated  into  conventional  control 
(procedures.  These  parameters,  then,  are  the  essential  character- 
I  istics  of  programs  which  are  considered  in  most  selection  processes 
1  and  also  form  the  basis  for  program  control. 

Despite  the  utility  of  time  and  cost  as  primary  program 

!  characteristics,  difficulties  encountered  in  the  administration  of 

i 
research  and  development  programs  indicate  that  these  parameters  may 

not  be  wholly  sufficient  for  the  purposes  intended.  To  this  extent, 

then,  the  following  discussion  will  examine  features  which  should  be 

added  to  control  procedures. 

Achievement  as  a  Parameter 

The  ultimate'-intent  of  any  research  and  development  effort  is 
the  realization  of  some  process,  procedure  or  product.  The  success 
of  the  effort  is  judged  by  a  comparison  of  the  actual  results  with 


Atkinson  and  Bobis,  "A  Mathematical  Basis  for  the  Selection 
of  Research  Projects,"  p.  3. 
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those  intended.   Planning  and  control  activities  associated  with  re- 
search and  development,  assume,  in  the  traditional  manner,  that  given 
research  efforts  can  be  characterized  by  measures  of  time  and  cost 
which  can  be  associated  with  the  subject  research  and  its  intended 
results.  As  was  discussed  in  the  introduction,  however,  notable 
failures  in  the  control  of  research  programs  have  been  experienced. 
This  usually  results  from  a  lack  of  correlation  between  research 
achievement  and  estimates  of  associated  time  and  cost. 

The  ineffectiveness  of  many  control  systems  can  be  attributed 
to  a  disregard  for  the  relationships  among  achievement,  time  and  cost. 
It  should  be  apparent  that  an  effective  control  system  must  yield 
valid  indications  of  the  ultimate  results  of  a  research  program  by: 
informing  the  manager  of  any  deviations  from  planned  values  of 
achievement,  time  or  cost;  indicating  corrective  actions  needed;  and 
communicating  these  indications  to  the  manager.   To  this  extent,  then, 
an  effective  control  system  must  merge  achievement  with  time  and  cost. 
However,  most  control  systems,  utilized  today,  separate  achievement 
reporting  from  cost  and  time  accounting  and,  therefore,  contribute 
to  ineffective  performance. 

Numerous  reasons  might  be  suggested  to  explain  the  observed 
disinterest  in  achievement  measurements  or  the  difficulties  en- 
countered when  such  measurements  have  been  attempted.  One  of  these 
reasons  is  that  it  is  frequently  difficult  to  analyze  achievement. 
For  some  development  projects,  achievement  indices  based  on  quality 
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and  quantity  of  accomplishments  can  be  devised.  However,  most  re- 
search projects  are  not  as  easily  handled  because  the  significance  of 
interim  findings  may  not  be  known  until  the  project  is  nearly  com- 
plete.  Consequently,  the  basic  mechanics  involved  in  implementation 
are  often  the  source  of  discouragement.  Many  of  the  difficulties 
encountered  in  attempting  to  quantify  progress  in  research  and  de- 
velopment result  from  a  literal-minded  approach  to  the  problem,  which 
is  reminiscent  of  conventional  production-type  operations. 

As  a  result  of  this  neglect  of  achievement  in  control  systems 
an  inordinate  amount  of  significance  is  generally  attached  to  project 
cost  overruns.  This,  in  turn,  contributes  to  the  introduction  of 

! unrealistic  reporting  of  results,  frequent  neglect  of  projects  which 
operate  within  the  budget  and  the  tendency  of  project  managers  to 

| expend  funds,  regardless  of  need,  in  order  to  avoid  budget  re- 

j duct ions  in  following  periods. 

In  summarizing  the  status  of  management  science,  as  it 
applies  to  research  efforts,  two  salient  features  warrant  emphasis. 
First  is  the  practice  of  selecting  and  funding  projects  primarily  on 
the  basis  of  a  priori   data;  that  is,  on  the  basis  of  initial  estimates 
of  cost,  time  .and  utility  of  the  product.  This,  however,  is  a  pe- 
culiar anamoly  when  one  considers  the  uncertainties  inherent  in  re- 
search and  development.  The  longevity  of  decisions  regards  selection 
and  funding  seems  quite  inappropriate  in  view  of  the  tenuous  nature 
of  the  informational  basis  for  these  decisions. 


1Ibid. ,  p.  42. 
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The  second  feature  of  the  process  concerns  the  nature  of  the 
inaccuracies  in  data.  The  estimates  required  by  conventional  se- 
lection and  funding  procedures  are  based  on  a  manager's  ability  to 
anticipate  the  performance  of  personnel  in  research  efforts.  How- 
ever,  in  many  research  programs  the  nature  of  the  forthcoming  work 
is  unknown.  Therefore,  it  is  virtually  impossible  to  ascertain        > 

! 

whether  the  talents  and  capabilities  of  a  research  team  are  appro- 
priate for  the  task.   Even  though  estimates  of  costs  for  equipment 
and  salaries  for  a  project  may  be  reasonably  accurate,  one  cannot 
determine,  on  an  a  priori   basis,  that  these  resources  will  be  used 
efficiently  and  yield  expected  result's. 


CHAPTER  III 

CONSTRUCTION  OF  THE  MODEL 

Having  examined  the  more  important  ingredients  of  the  problem 
of  program  selection  and  control,  attention  will  now  be  focused  on  the 
I  procedures  which  are  advocated  by  this  paper.   The  procedures  and 
I  techniques,  which  have  been  selected  for  use,  constitute  the  model 
'which  is  the  basis  of  the  quantitative  method.   The  two  major  cora- 
[ponents  of  the  model,  dealing  with  selection  and  control,  are  dis- 
j cussed  separately  for  they  are  two  relatively  distinct  operations. 

Project  Selection 

The  process  of  project  selection,  as  the  preceding  discussion 
j has  related,  and  as  is  shown  in  Figure  2,  involves  the  measurement  of 
project  utility  with  consideration  given  to  established  objectives, 
and  taking  into  account  the  effect  of  interdependence  among  projects. 

As  was  pointed  out,  quantitative  techniques  are  not  employed 
to  any  extent  until  the  formulation  of  operational  objectives.  Con- 
sequently, this  discussion  will  assume  that  preliminary  planning  has 
been  accomplished  to  the  extent  of  having  established  operational 
objectives.  Therefore,  what  remains  to  be  shown  is  the  method  of 
completing  the  selection  process  on  the  basis  of  these  operational 
objectives,  project  utility,  interdependence  and  certain  aspects  of 
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funding. 

Utility  Analysis 

In  selecting  a  quantitative  method  for  utility  analysis,  one 
must  first  decide  how  interdependence  is  to  be  incorporated.   This 
does  not  result  from  any  particular  theoretical  requirement,  but,  is 
simply  a  starting  point  which  facilitates  construction  of  the  model. 

The  preceding  chapter  discussed  two  approaches  to  interde- 
pendence; that  used  in  PATTERN,  which  is  based  on  the  direct  measure- 
ment of  cross-support  among  projects,  and  that  used  in  PROFILE,  which 
uses  indirect  measurement  methods.   Either  of  these  approaches  are 
adequate  for  purposes  of  project  selection.  Therefore,  the  basic 
technique  of  the  PROFILE  approach  is  adapted  since  it  is  more  easily 
implemented. 

The  PROFILE  approach  relates  project  characteristics  to  re- 
search objectives.   However,  in  actual  application  situations  at  the 
laboratory  level,  the  relationships  between  projects  and  general 
objectives  may  be  confused  by  intermediate  requirements.  Moreover, 
relating  project  characteristics  directly  to  general  objectives,  which 
are  usually  strategic  and  of  managerial  nature,  does  not  correspond 
to  the  process  described  in  the  preceding  chapter,  and  shown, 
graphically,  in  Figure  2.   Therefore,  the  method  developed  here  will 
utilize  requirements  prescribed  at  the  laboratory  level  in  accordance 
with  operational  objectives. 

The  process  of  relating  research  projects  to  research  re- 
quirements, for  the  purpose  of  project  comparison,  requires  that  the 
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1  subject  requirements  be  expressed  in  terms  that  are  applicable  to  all 
projects;  otherwise  the  continuity  of  the  comparison  would  be  destroyed. 
;  In  order  that  this  requirement  of  continuity  may  be  realized,  an 
j  added  measure  is  interjected.   This  added  measure  is  expressed  in 
terms  of  scientific  disciplines  which  relate  to  both  projects  and 
requirements.   The  desired  relationships  are  then  obtained  by  de- 
scribing requirements  in  terms  of  the  scientific  disciplines,  or 
technologies,  which  must  be  supported,  and  by  describing  projects  in 
terms  of  the  technologies  to  which  they  relate.  The  method  de- 
scribed is  shown  graphically  in  Figure  4. 

To  describe  the  process  of  relating  projects  to  sciences,  con- 


sider three  projects,  Pj,  P2  and  P.,,  which  contribute,  in  various  com- 

2 
binations  to  four  sciences,  S,,  S2,  S^  and  S^ .   Next,  assume  that  Pj 

supports  S  and  S2,  ?2   supports  S2,  S,  and  S,,  and  P  supports  S^  and 

S  .   It  is  now  possible  to  c 

structing  a  transfer  matrix 


S  .   It  is  now  possible  to  quantify  these  relationships  by  con- 


A=  (ai;j) 


where 


1,  of  P-  supports  Sj  ; 
10,  if  P^  does  not  support  S  •  . 


B.  V.  Dean,  "A  Research  Labortory  Performance  Model,"  IEEE 
Transactions  on  Engineering  .Management,  EM-14  (March,  1967),  pp.  44-45. 

2Ibid. 
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FIGURE  4 


DIAGRAM  OF  PROJECT,  SCIENCE 
AND  REQUIREMENT  RELATIONSHIPS 
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Source:   Dean,  "A  Research  Laboratory  Performance  Model,"  p.  44 
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If  the  project-science  relationships  are  presented  as  in  Table  5,  it 
is  apparent  that 


(III-D 


The  relationships  between  sciences  and  research  requirements 
can  be  similarly  described  using  the  transfer  matrix 


1 

1 

0 

0 

0 

1 

1 

1 

1 

0 

1 

0 

B=  (bjk) 


where 


»jk  = 


1,  if  a  development  in  Sj  is  required  to  satisfy  R^; 
,0,  otherwise. 


If  the  science-requirement  relationships  are  assumed  to  be 
those  shown  in  Table  6,  then 


B  = 


1 

0 

0 

0 

1 

1 

1 

0 

1 

1 

1 

1 

(III-2) 


On  the  basis  of  the  preceding  analysis,  it  is  now  possible  to 
determine  the  relationships  between  projects  and  requirements,  which 
was  the  original  intent.   Recall  that,  if  project  i  contributes  to 
science  j,  then 
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TABLE  5 

PROJECT- 

SCIENCE  RELATIONSHIPS 

Sciences 

Pl 

Si 

s2   s3 

S4 

1 

1    0 

0 

Projects 

P2 

0 

1    1 

1 

P3 

1 

0    1 

0 

~- -^ 

TABLE  6 

SCIENCE- REQUIREMENT  RELATIONSHIPS 


Requirements 

si 

Rl 

R2   R3 

1 

0    0 

S2 

0 

1    1 

Sciences 

S3 

1 

0    1 

s4 

1 

1    1 
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"Ij  ■ l- 

Likewise,  if  a  development 

in 

science 

j  is  necessary 

to 

satisfy 

requ- 

ireraent  k,  then 

b.u  =  1. 

Jk 

Consequently,  if  project  i 

ultimately 

contribut 

es  to 

requirement 

k, 

then 

aijbjk  =  1. 

j  Likewise,  if  either 

aij  =  °.  or  bjk  =  ° 

then  project  i  is  not  related 

to  requirement  k, 

and 

a.  .b-,  =  0. 
ij  3k 

It  should  be  observed  that  a  given  project  can  contribute  to 
'some  requirement  in  more  than  one  manner.   That  is,  a  project  may 
['support  several  different  sciences,  which  each  contribute  to  the  same 

i 

j requirement.   For  purposes  of  selection,  it  is  necessary  to  determine 
[the  number  of  ways  in  which  each  project  does  contribute  to  each  re- 
quirement.  If  c^  is  identified  as  the  number  of  ways  in  which  project 
i  supports  requirement  k,  the  c^k  can  be  computed  from 


Cik  =  I   aijbjk  ' 


If  C  is  identified  as  the  matrix  of  c,  ,  then 
—  ik* 

C  =  A  B 

To  illustrate  the  application  of  this  analysis,  consider  the 
values  of  A  and  B  given  in  equations  (III-l)  and  (III-2).   For  these 
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values, 


A  B 


1 

1 

0 

0 

0 

1 

1 

1 

1 

0 

1 

0 

1 

0 

0 

0 

1 

1 

1 

0 

1 

1 

1 

1 

c  = 


1  1  1 

2  2  3 
2  0  1 


(III-3) 


The  significance  of  £  can  be  shown  by  an  examination  of  some  of  the 
elements,  c^.   For  instance, 

cll  =  c12  =  c13  =  ! 
indicates  that  project  1  contributes  to  all  three  requirements  in  one 

way.  The  fact  that 


:23 


=  3 


however,  indicates  that  project  2  contributes  to  requirement  3  in 
three  different  ways.   This  is,  indeed,  the  case,  via  sciences  2,  3 
and  4. 

The  structure  of  the  model,  thus  far,  and  the  method  of 
analysis  employed,  enables  the  ready  determination  of  project  utility 
to  the  extent  that  the  utility  is  a  function  of  the  number  or  require- 
ments supported.  This,  however,  is  seldom  the  criterion,  for  some  re- 
quirements are  usually  more  important  than  others.   Indeed,  this 
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would  likely  be  the  case  when  considering  operational  objectives. 

When  objectives  or  requirements  are  examined,  it  is  fairly 
easy  to  rank  them  in  an  order  of  importance  or  desireability.  This 
was  demonstrated  in  the  preceding  chapter  when  the  parameters  of  stor- 
age batteries  were  considered.   Any  of  several  rating  techniques  may 
be  utilized,  depending  on  the  nature  of  the  requirements  being  con- 
sidered.  Regardless  of  how  requirements  are  assessed,  the  result  will 
i be  a  value  function,  V.   Assume,  for  example  that  a  research  admin- 
istrator had  examined  the  research  requirements  R. ,  R2  and  R3,  and 
j rated  each  on  a  0  to  10  scale,  and  that  the  results  of  his  evaluation 
were 

v(Rx)  =  10, 
v(R2)  =  2 
and    v(R3)  =  4  . 


In  order  to  ensure  the  uniformity  of  the  selection  logic,  these 
ratings  must  be  normalized  to  satisfy 

I  Vi  =  X. 

i 

After  normalization,  it  is  possible  to  determine 


V  =(Vi)= 
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Having  determined  the  value  of  the  research  objectives  or  requirements, 

i 

it  is  then  a  simple  task  to  evaluate  the  research  projects.   Recall 

i 

that  £,  as  shown  in  equation  (III-3),  relates  projects  to  requirements 
in  terms  of  the  number  of  requirements  supported.   Having  determined  V, 
one  can  then  ascertain  the  value  of  projects,  r. ,  in  terms  of  the  values 
of  the  several  objectives,  v^,  from 

R=  <ri)  =  £  v.  ciii-4) 

This,  for  the  example  used  previously  is  found  to  be 

1 


(III-5) 


| which  shows  the  relative  values  of  the  three  candidate  programs. 

Success  in  using  analysis  techniques,  such  as  those  employed 
in  PROFILE,  is  dependent  on  the  degree  of  orderliness  achieved  in  the 
process  of  determining  the  possible  combinations  of  project-science 
and  science-requirement  relationships.   In  order  to  illustrate  an 
approach  which  has  some  likelihood  of  success,  a  method  developed  for 
the  Naval  Material  Command,  Washington,  D.  C,  is  presented  in 
Appendix  A. 

Expenditure  Analysis 

After  having  computed  the  value  of  alternative  projects,  it 
is  necessary  to  examine  project  funding  so  that  a  cost-benefit 
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. 

1 

S comparison  can  be  conducted.   The  primary  interest,  in  this  regard 
jlies  in  the  relationship  between  expenditure  and  achievement.   Mien 


the  research  manager  is  contemplating  the  allocation  of  funds,  this 
relation 
I  funding. 


•relationship  is  needed  to  determine  the  desired  level  of  project 


4 


Over  time,  the  distribution  of  expenditures  for  research  and 
development  projects,  is  generally  expected  to  be  similar  to  one  of 
two  likely  patterns.   The  first  typical  pattern  shown  in  Figure  5a, 
is  the  decaying  exponential;  approximated  by 

fix)   =  e"ax  .  (III-6) 

This  type  of  distribution  characterizes  a  project  which  commences  with 
large  initial  outlays,  with  a  subsequent  decline  in  expenditures.   The 
second  typical  pattern,  shown  in  Figure  5b,  represents  projects  which 
commence  more  slowly,  usually  in  some  small  exploratory  effort,  and 


'build  up  to  some  maximum  expenditure  which  is  then  followed  by  decline. 

i 
Although  these  expenditure  distributions  are  of  interest  for 


j  budgetary  use,  it  is  the  cumulative  distribution  which  is  of  greater 
interest  for  purposes  of  project  selection.   As  indicated  in  Figure  6a, 

J  the  cumulative  distribution,  corresponding  to  the  expenditure  pattern 
of  Figure  5a,  is  represented  by  a  function  of  the  form 

f(x)   =  1  -  e"ax  .  (IH-7) 

Similarly,  the  distribution  of  Figure  5b  relates  to  the  general  S 


IB.  V.  Dean  and  S.  S.  Sengupta,  "Research  Budgeting  and 
i  Project  Selection,"  IEEE  Transactions  on  Engineering  Management, 
j  EM-9  (December,  1962),  p.  169.  "       " 
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FIGURE  5 
TYPICAL  EXPENDITURE  DISTRIBUTIONS 


time  (x) 
(a) 


Source:   Dean  and  Sengupta,  "Research  Budgeting  and  Project 
Selection,"  p.  169. 
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FIGURE  6 
CUMULATIVE  EXPENDITURE  DISTRIBUTION'S 


time    (x) 
(a) 


time    (x) 


Source:     Original;   derived  from  integration  of  distributions   in 
Figure   5. 
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curve  shown  in  Figure  6b.   The  distributions,  however,  relate  cost 
to  time.   What  is  actually  of  interest  is  the  relationship  between 

:  cost  and  achievement. 

The  impact  on  project  achievement  of  variations  in  funding 

I  can  be  determined  by  considering  the  progress  of  a  typical  project  for 

different  funding  levels.   Figure  7  shows  project  progress  for  four 

different  funding  levels.  The  four  levels  illustrated  commence  with 
I 
level  1,  which  is  one-fourth  of  the  satiation  level,  M,  and  increase 

t 
in  one-quarter  increments  until  M  is  reached. 

To  determine  the  cost-achievement  relationship,  one  simply 

]  takes  a  given  time,  t0,  and  compares  cost- achievement  points  for  that 

time.   The  results  of  such  an  examination  are  shown  in  Figure  8.   As 

|  is  indicated,  for  the  achievement-time  relationship  of  Figure  7,  the 

i 

.'  resulting  cost-achievement  relationship  is  the  familiar  exponential 

! 

|  as  given  in  equation  (III-7).  However,  the  cost-achievement  function, 

i  under  other  circumstances,  may  be  similar  to  Figure  8b.   The  ultimate 

i 

conclusion,  then,  is  that  achievement,  as  a  function  of  cost,  is 


similar  to  either  the  decaying  exponential,  or  the  S-curve. 

Quantifying  the  cost-achievement  function  of  research  projects 
can  be  simplified  even  further.  As  shown  in  Figure  8,  the  two  types 
of  distribution  are  essentially  the  same  above  the  achievement  level 
corresponding  to  critical  cost.   Since  management  is  not  likely  to 
be  interested  in  funding  levels  below  the  critical  level,  the  two 


^V.  E.  Souder,  "Planning  R  5  D  Expenditures  With  the  Aid  of  a 
Computer,"  Budgeting,  March,  1966,  p.  27,  or  Ay res,  Technological 
Forecasting  and  Long-Range  Planning,  p.  177. 
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FIGURE  8 
COST- ACHIEVEMENT  RELATIONSHIP 
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Source:   Soudcr,  "Planning  R5  D  Expenditures  with  the  Aid  of  a 
Computer,"  p.  27. 
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:  distributions  can  be  modeled  by  the  expression  for  Figure  8a,  as 

|  given  in  Equation  (II 1-7) . 

Having  determined  a  form  for  cost-achievement,  it  is  next 

!  necessary  to  find  the  parameters  for  the  selected  exponential.   As 

|  would  be  presumed,  the  exponential  is  a  function  of  the  proposed  fund 

I  allocation  for  project  i,  x. .   In  keeping  with  convention,  x.  is 

1  1 

normalized  by  the  function's  time  constant, 

1 

a  ' 

which  is  that  value  of  x.  at  which  63  per  cent  of  maximum  achieve- 
ment is  realized.  This  notation  is  shown  in  Figure  9. 

Since  the  functional  form  of  cost-achievement  has  been 
uniquely  specified,  the  relationship  for  any  particular  project  can 
be  found  explicitly  once  any  single  data  point  is  obtained.  The 
procedure  for  this  becomes  apparent  when  one  realizes  that  the  1.0 
achievement  level  in  Figure  9  corresponds  to  maximum  (or  unlimited) 
funding.   Consequently,  achievement  is  measured  relative  to  the 
possible  maximum.  An  achievement  estimate  can  then  be  obtained,  for 

a  particular  proposed  expenditure.   By  examining  a  research  forecast, 
I 
it  is  possible  to  compare  the  expected  advance  in  the  achievement 

!  parameter,  for  the  proposed  expenditure,  with  the  advance  that  might 

be  expected  from  a  maximum  expenditure.   Figure  10  shows  the  appli- 

i  cation  of  this  method  for  a  research  project  concerning  high  strength 

I  resins.    For  this  research  project,  then,  an  achievement-cost  curve 

i  


U.S.,  Department  of  the  Navy.  Naval  Applied  Science  Labora- 
|  tory,  Technological  Forecast  in  Depth;  Organic  Materials,  196S-1988, 
I  Lab.  Project  950-100,  Final  Report  (1968),  p.  18. 
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would  be  determined  by  plotting  x.  on  the 

abscissa,  a  ./au  on  the 

ordinate,  solving  for  t  from 

*i 

!*i  =  1  -  e"  T 

aM 

and  completing  the  curve  for  the  value  of 

! 

t  thus  obtained. 

Optimization  of  Selection 

As  the  result  of  utility  analysis 

and  expenditure  analysis, 

1 

each  project  can  be  characterized  quantitatively  by  combining 

!  equations  (I I 1-4)  and  (III-7)  to  obtain  a  profit  function  for  each 

I 
project  in  the  form 

Pi  -  ri  (1  -  e  Ti)  (III-8) 

I 

|  The  objective  of  the  optimization  procedure,  then,  is  to  maximize 

P  =  I   Pi  (IH-9) 

i 

subject  to  a  budget  constraint  of 

l  H  <   B  ,  Xi  >.0  .  (111-10) 

i 

Presuming  that  the  intent  is  to  expend  the  budgeted  amount,  this 

constraint  becomes 

B  =  I  H    ,   x^O.  (III-ll) 

i 

The  task  of  optimizing  equation  (II 1-9)  can  be  accomplished  by  using 
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the  technique  of  the  Lagrange  multiplier.   To  implement  this 
technique,  let 

Q  =  P  -  XB 

_xi 

•   =  I   [  ri    (1  -  e  ^  -  XXjL  ]  (111-12) 

i 

where  X  is  some  constant  (the  Lagrangian  multiplier).   Now,  since  Q 

j  differs  from  P  by  only  a  constant,  XB,  then,  a  maximum  for  Q  will 

i 

;  correspond  to  a  maximum  for  P.   To  find  the  maximum  for  Q,  compute 


»*,  ° 


r,  -2 


1 

! 

*r* 

li  - 

V  =  0 

• 

(111-13) 

I 

1  This 

1 

is  ensured  to 

be  a 

maximum, 

since 

32^ 

.  ri 

xi 

e'Ti 

<  0  . 

3xJ 

xi 

From 

equation 

xi 

e"Ti: 

(III 
^i 

-13), 

then, 

(111-14) 

xi  = 

rj.  In 

xTi 

)ff  and  E. 

S.  Sokolnikoff, 

(III-1S) 

Higher  Mathematics 

h.    S 

.  Sok 

Dlnikc 

for 

Engineers 

and 

Physicists  (New 

York : 

McGraw-Hill 

Book  Company, 

Inc. 

,  1941),  pp.  165-67 

..     I 
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Now, 

as  was  provided  in  equation  (III-ll),  the 

only  solutions  to 

be 

admitted  are  those  for  which 

xi  L  °  • 

This 

in  turn  requires  that  the 

argument  of  the 

logarithm  in  (Ill- 

! 

15) 

j 

be  > 

1.  These  constraints  can 

be  re-stated  by 

acknowledging  that 

for 

1 

any  i 

! 

,  either 

(III-16.1) 

! 

i 
i 

i 
I 

or 

! 

^ 

(III-16.2) 

i 

I 

! 
1 

i 

! 

i  To  explain  the  significance  of 

equation  (111-16),  assume  that  of 

! 

a 

;  total 

of  n  possible  projects,  some  sub-set,  a, 

of  the  n,  constitutes 

i  t 
;  those 

s  projects  which  will  be  funded  for  optimum  return.   For  each 

of 

!  the  projects  in  a,  since  x^  is 

non-zero,  then 

! 

I 

Of  course,  for  all  projects  not  in  o 

xi  =  0  . 

In  view  of  this,  equation  (III- 

■11)  becomes 

B  -  I  H   . 

(111-17) 

0 

-t 

Now, 

combining  equation  (111-17)  with  equation 

(I II- 15)  ,  one  obtains 

1 

1 
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v       ,     ri 
B  =    ITi  ln  XTi 
a 

(111-18) 

which  can  be  re-written  as 

A  =  exp  < 

fiw- 

■    B 

> 

(111-19) 

L              0 

' 

It  is  evident  that  the  value  of  X  is  a  function  of  the  projects 
selected  for  inclusion  in  a.   Consequently,  in  order  to  implement  this 
optimization  procedure,  some  approach  is  needed  which  avoids  the 
necessity  of  considering  the  multitude  of  possible  combinations  for  o. 

To  reduce  the  effort  required  in  seeking  o,  the  following 
algorithm  can  be  employed.  First,  the  candidate  projects  must  be 
arranged  in  descending  order  of  the  cost-benefit  factor, 


I  In  other  words,  arranged  such  that 


r,   r, 


(111-20) 


Next,  a   is  assumed  to  consist  of  project  1  and  project  2,  and  X  is 
computed  from  equation  (I 11-19) .   Then,  equation  (III-16.2)  is  invoked 
for  i  =  1  and  i  =  2.   If  equation  (III-16.2)  is  satisfied  in  each  case, 
then  o  is  expanded  to  include  project  3.  The  subset,  o,  is 
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I 

J 

j  progressively  expanded,  with  X  being  re-computed  and  checked  at  each 

| 

!  iteration.  The  iterations  cease  when  a  project,  k,  is  introduced  into 

j  the  subset  such  that  equation  (III-16.2)  is  violated.   Having  reached 

j  this  point,  it  is  determined  that  a  consists  of  the  first  k-1  projects 

in  the  ordering  of  equation  (111-20).   Then,  X  is  computed  from 

equation  (III- 19) ,  including  the  selected  k-1  projects.   Finally, 

for  this  value  of  X,  the  fund  allocation  for  each  of  the  selected 

projects,  x.,  is  determined  from  equation  (111-15).  This  algorithm 

is  shown,  in  flow- chart  form,  in  Figure  11. 

Should  a  consist  only  of  project  1,  the  combination  of 

equations  (II 1-15)  and  111-19)  reveals  that 

x.  =  B  , 
l 

;  as  one  would  expect.   If,  then,  project  2  is  added  to  a,  X  increases 
;  as  per  equation  (II I- 19) .   Likewise,  x,  is  de-creased  to  some  value 
! less  than  B,  with  the  balance  being  allocated  to  x^: 

Xj  <  B 

x2  =  B  -  x<l  . 

•  As  X  increases  with  each  addition  of  a  project  to  a,  it  is  compared, 
j  via  equation  (III-16.2),  with  successively  smaller  values  of 


U 
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FIGURE  11 
SELECTION  ALGORITHM 
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ri 

Consequently,  if  one  were  to  plot  the  values  of  X  and  the  limiting  — 

Ti 
for  the  series  of  o's,  the  resulting  locus  as  illustrated  in  Figure  12, 

would  demonstrate  the  determination  of  a. 

Project  Control 
•  Chapters  I  and  II  examined  various  aspects  of  management  plan- 
ning and  control  as  they  relate  to  the  administration  of  research  and 
S  development  activities.   As  was  illustrated  in  Figure  1,  the  essential 
!  ingredients  of  the  control  process  are  the  comparison  of  actual  and 
!  anticipated  performance  and  an  analysis  of  observed  variances.   Though 
•  this  description  of  control  is,  at  face  value,  no  different  from 
:  control  in  conventional  management  systems,  the  significance  of  con- 
j  trol  in  research  and  development  activities,  in  terras  of  its  im- 


portance for  effective  and  efficient  operations,  is  substantially 

1 
greater.   As  has  been  mentioned  before,  control  in  most  enterprises 

relates  to  budgetary  control  of  operations.  Traditionally,  this  type 

of  control  has  been  successful.   The  degree  of  success  that  has  been 

realized  is  largely  attributeable  to  the  close  correlation  between 

resources  expended  and  results  realized.   Such  a  nexus  is  obvious  in 

practices  which  include  instruments  like  standard  cost  schedules 

for  production  activities.   Even  less  structured  business  operations, 

such  as  advertising,  are  ammenable  to  analysis  on  the  basis  of  results 

2 
realized  from  given  expenditures. 


Souder,  "Experiences  with  an  R  5  D  Project  Control  Model,"  p. 39 

2 

Jay  W.  Forrester,  Industrial  Dynamics  (New  York:  The  M.  I.  T. 

Press  of  the  Massachusetts  Institute  of  Technology  and  John  Wiley  & 
Sons,  Inc.,  1961),  pp.  199-207. 
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As  was  discussed  in  Chapter  II,  personnel  associated  with  re- 
search and  development  activities  have,  in  several  studies,  demon- 
strated an  ability  to  estimate  anticipated  expenses.  However,  other 
similar  attempts  have  resulted  in  notable  failure.  The  essential 
characteristic  of  research  and  development  which  prompts  this  dichotomy 
is  the  uncertain  correlation  between  the  activities  for  which  cost 
estimates  are  prepared  and  the  research  advances  which  are  expected 
{to  result  from  these  activities.  The  supervisor  of  a  research  team 
I  may  be  quite  capable  of  estimating  costs  for  a  proposed  project.  How- 
lever,  the  accuracy  of  his  estimates,  regards  the  effectiveness  of  his 
i organization  in  persuing  and  achieving  technological  advance,  may  be 
{inadequate.   As  a  result,  control,  in  research  and  development,  must 

'monitor  not  only  expenditures,  but,  also,  the  achievements  accruing 

j 

1 

;  from  these  expenditures. 

The  Role  of  Prior  Estimates 


A  prior  estimate  is  a  statement  of  anticipated  circumstances 
associated  with  some  future  effort.  The  fact  that  prior  estimates 
concern  future  efforts  is  essential,  since  the  estimate  is  presumed 
to  exclude  any  information  of  the  actual  prosecution  of  the  effort. 
In  this  respect,  then,  an  annual  budget  is  a  prior  estimate  since  it 
concerns  activities  which  are  yet  to  commence. 

Prior  estimates  are  common  to  the  vast  majority  of  managerial 
endeavors.   Besides  financial  budgets,  such  estimates  include  production 


Jantsch,  Technological  Forecasting  In  Perspective,  pp.  53-6D. 
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schedules,  personnel  projections,  material  schedules,  and  the  like. 
These  estimates  derive  from  objectives  which  are  the  results  of  plan- 
ning.  Most  such  estimates,  for  conventional  commercial  concerns,  are 
fairly  objective  from  the  stand  point  that  the  estimates  relate  to 
activities  of  very  similar  nature.  Therefore,  these  estimates  are 
frequently  obtained  by  using  simple,  extrapolation  techniques. 

Estimates  for  research  and  development  activities  are  also 

{obtained,  in  most  instances,  by  extrapolation  techniques.   Indeed, 

i 

;  one  of  the  most  common  techniques  employed  in  forecasting,  or  esti- 

•  mating,  is  that  of  trend  extrapolation.   The  use  of  these  common 

|  extrapolation  techniques  presumes  the  existence  of  a  substantial  a- 

I 

I  mount  of  continuity  between  the  past  and  the  immediate  future.  As 

;  has  been  emphasized  in  preceding  discussion,  such  a  presumption,  re- 
lating to  research  and  development,  is  inherently  uncertain.   Specifi- 
cally, two  questions  must  be  addressed  when  extrapolation  is  attempted 

1)  .  What  are  the  criteria  for  choosing  curves  to  extrapolate? 

2)  When  is  it  safe  to  extrapolate  a  rate  of  change  naively, 
and  when  must  points  of  inflexion,  or  changes  in  the  rate  of  change, 
be  anticipated? 

These  are  two  questions  that  are  not  easily  answered. 

In  attempting  to  minimize  the  impact  of  poor  results  from 


James  R.  Bright,  Technological  Forecasting  for  Industry  and 
Government—Methods  and  Applications  (Englewood  Cliffs:   Prentice-Hall, 
Inc.,  1968),  pp.  56-109. 

2Ibid. ,  p.  77. 
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extrapolation  methods,  many  management  systems,  for  research  and  de- 
velopment, implement  measures  of  the  uncertainty  involved,  in  the 
form  of  probabilities  of  success.   However,  these  estimates  are  still 
prior  estimates,  and  are  usually  obtained  by  some  sort  of  extrapola- 
tion. The  difficulties  associated  with  such  efforts  have  been  dis- 
cussed in  Chapter  I. 

The  over-riding  consideration  that  must  be  recognized  when  ad- 
dressing the  topic  of  prior  estimates,  is  that  the  utility  of  such 
techniques,  as  they  have  been  applied  to  conventional  management 

j systems,  is  limited.   Even  attempted  modifications,  which  have  been 

I 

| initiated  as  a  result  of  this  limited  utility,  are  difficult  to  im- 

I 

I 

iplement,  and  frequently  have  the  same  inherent  shortcomings  possessed 

by  the  methods  which  they  seek  to  improve. 

From  the  preceding  analysis,  one  should  not  conclude  that 
prior  estimates  have  no  valid  role  in  research  and  development  manage- 
ment. On  the  contrary,  they  are  essential.  All  quantitative  selec- 
tion techniques  initiate  from  prior  estimates.   It  should  be  obvious, 
that,  since  selection  necessarily  takes  place  before  any  activity 
commences,  only  prior  estimates  are  available.   Project  control,  how- 
ever, is  a  continuing  process  throughout  the  life  of  the  project.   It 
is  reasonable,  therefore,  to  contend  that  one  need  not  and  should  not 
rely  solely  on  prior  estimates  for  purposes  of  control.  The  remainder 
of  this  chapter  will  be  devoted  to  the  development  of  a  method  which 
augments  prior  estimates  and  affords  increased  vitality  for  methods 
of  controlling  research  and  development  activities. 
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Description  of  the  Process 

The  preparation  of  a  proposal  for  a  research  project  necessari- 
ly commences  with  the  development  of  a  scenario.   This  scenario  is  a 
time-ordered,  episodic  sequence  of  events  which  are  logically  related 
to  one  another  and  portray  the  intended  progress  of  the  project.   In 
informal  organizations,  scenario  preparation  may  be  nothing  more  than 
the  act  of  mentally  planning  the  effort.  However,  in  most  situations, 
j which  require  the  submittal  of  proposals  in  a  formal  fashion,  the 
i  scenario  will  be  explicit  in  nature  and  set  forth  in  a  published  form. 
j  Procedures  for  the  preparation  of  proposals,  and  attendent  scenarios, 
:  vary  greatly  among  research  organizations.  The  particular  approach 
!  described  here  is  one  common  to  some  of  the  laboratories  of  the  U.S. 
;  Navy. 

To  illustrate  the  development  of  a  scenario,  it  is  advanta- 
geous to  examine  a  sample  project.   For  this  purpose,  consider  the 

i 

j  task  of  developing  a  technique  for  the  optimal  design  of  super-con- 
ducting electrical  machines,  intended  for  solution  by  electronic  com- 
puter. The  reason  this  task  would  be  considered  a  research  and  de- 
velopment effort  derives  from  the  fact  that  super-conducting  machines 
do  not  conform  to  conventional  electro-magnetic  theory  and  established 
design  procedures  have  not  been  developed.  Moreover,  since  past  design 
and  construction  endeavors  have  been  limited  to  a  few  prototypes,  there 
exists  no  consensus  concerning  an  accepted  theoretical  model. 


^The  method  discussed  is  one  obtained  during  the  period  of 
time  when  the  author  was  Program  Officer  at  the  Naval  Applied  Science 
Laboratory,  Brooklyn,  New  York. 


86 

A  scenario  for  this  research  task  consists  of  the  following 

1 

events: 

1)  Determine  appropriate  parameters  for  the  model. 

2)  Develop  magnetic  field  equations  utilizing  the  parameters 
selected. 

3)  Determine  operational  and  theoretical  constraints. 

4)  Develop  design  equations. 

5)  Develop  optimization  method. 

6)  Write  machine  program. 

7)  Complete  analysis  of  test  designs. 

The  initial  opinion  one  is  likely  to  formulate,  concerning 
such  a  scenario,  is  that  it  is  simply  an  application  of  conventional 
scheduling  techniques,  such  as  PERT.   To  an  extent,  such  an  observa- 
tion is  correct,  for  both  serve  to  indicate  project  progress,  or 
j  schedule.   However,  it  should  be  observed  that  the  scenario  is  a  se- 
quential ordering  which  does  not  include  any  activities  conducted  con- 
currently. On  the  contrary,  the  events  in  a  scenario  can  be  arranged 
only  in  serial  order.  Consequently,  completion  of  a  scenario  event 
indicates  explicitly  some  degree  of  project  progress.  Nonetheless, 
PERT  network  analysis  is  of  sufficient  similarity  to  the  scenario  to 
deserve  further  examination.  The  results  of  such  an  examination  has 


D.  P.  Greeneisen,  "A  Design  Program  for  Superconducting  Elec- 
trical Machines"   (unpublished  M.  S.  thesis,  Massachusetts  Institute  of 
Technology,  1968),  pp.  7-21. 

2 

C.  McMillan  and  R.  F.  Gonzalez,  Systems  Analysis  —  A  Computer 

Approach  to  Decision  Models  (Homewood,  Illinois:   Richard  D.  Irwin,  . 
Inc.,  1968),  p.  292. 
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revealed  that  PERT  techniques  can  be  used  in  more  complex  projects 
where  a  single  scenario  may  be  difficult  to  identify.   However,  PERT 
analysis  will  be  useful  only  in  those  instances  where  projects  can 
be  sub-divided  into  appropriately  isolated  subsidiary  networks  which 


meet  certain  criteria.   In  the  final  analysis,  thefore,  whether  one 


considers  the  scenario  to  be  a  special  case  of  the  general  PERT 

technique  is  relatively  immaterial.  The  only  essential  characteristics, 

'which  must  be  maintained,  are  those  of  sequentiality  in  the  ordering 

|  of  events,  and  the  selection  of  events  such  that  technological  progress 

;  is  represented.  The  importance  of  the  second  requirement  will  become 

obvious  as  the  description  proceeds. 

After  determining  the  events  of  the  scenario,  described  above, 

I  it  is  necessary  to  prescribe  a  procedure  for  utilizing  the  data  con- 
|  i 

tained  therein.  Towards  this  end,  it  is  beneficial  to  use  the 

i  scenario  events  as  milestones  for  the  computation  of  time  and  cost 

;  projections. 

The  development  of  a  time  projection  may  come  about  in  one  of 

two  possible  ways.   First,  time  until  completion  may  be  a  given  con- 

I  straint  on  the  problem.   In  this  case,  it  is  necessary  to  determine  a 

!  manning  level  distribution  which  will  be  adequate  to  meet  the  time  re- 

! 
quirement.  The  second  possibility  is  that  personnel  resources  may  be 

a  governing  constraint.  This,  in  turn,  would  require  an  estimate  of 


Rosenbloom,  "Notes  on  the  Development  of  Network  Models  for 
Resource  Allocation  R  £  D  Projects,"  p.  62. 

R.  E.  Beckwith,  "A  Cost  Control  Extension  of  the  PERT  System," 
IRE  Transactions  on  Engineering  Management,  EM-9  (December,  1962),  p. 
147. 
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the  time  needed  for  completion  with  the  given  personnel  available.   In 

actuality,  a  third  possibility  exists,  for  which  neither  personnel 

! 
available  nor  time  is  specified.   In  this  instance,  a  trade-off  be- 
tween the  two  would  be  sought.   The  solution  for  the  optimal  trade-off 
is  a  substantial  task  in  itself,  and  will  not  be  pursued  further. 

Regardless  of  which  one  of  the  two  situations  occurs,  the 

i 
determination  of  manpower  or  time  requirements,  from  the  given  con- 

jstraint,  entails  an  estimate  of  personnel  capability  in  terms  of  the 

!  i 

'particular  project  under  consideration.   Any  research  organization 
I  contains  a  great  variety  of  skills  and  many  levels  of  efficiency. 
Certain  personnel  may  have  had  experience  which  would  be  particularly 

applicable  to  the  project  under  consideration.   If  these  personnel  are 

f 

i available,  completion  of  the  project  will  be  expedited.   If,  on  the 

I  other  hand,  some  novice  talent  is  to  be  used,  added  time  for  completion 

j 

■must  be  allowed.   The  primary  consideration  which  is  being  addressed, 

J  here,  is  that  time  estimates  are  functions  of  the  administrator's  ex- 
pectations regards  the  technological  demands  of  the  research  project, 
and  his  estimates  of  the  capability  of  a  given  research  team  to  cope 
with  these  demands. 

In  addition  to  the  preparation  of  a  time  estimate,  the  scenario 
will  form  the  foundation  of  a  cost  estimate.   Ideally,  the  scenario 
will  be  composed  of  events  which  represent  stages  of  the  project  which 
are  reasonably  different  in  terms  of  the  activity  involved.   For 


For  further  consideration  of  this  question  see  R.  A.  Goodman, 
"Organization  and  Manpower  Utilization  in  Research  and  Development," 
IEEE  Transactions  on  Engineering  Management,  EM-IS  (December,  1968), 
p.  198. 
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instance,  in  the  example  scenario  which  has  been  presented,  the  deter- 
mination of  operational  and  theoretical  constraints  and  the  development 
of  design  equations  are  related  to  the  extent  that  one  is  required  for 

the  accomplishment  of  the  other,  and  both  are  dealing  with  the  same 

I 
subject  matter.   However,  the  actual  activity  involved  in  each,  in 

terms  of  solution  methods,  reference  material  and  theoretical  con- 

i 

siderations,  will  be  different.   This  independence  of  scenario  events 
I  is  not  a  theoretical  imposition,  but,  simply  serves  to  facilitate 

!  •  J 

!  time  and  cost  analyses. 

Cost  estimates  for  scenario  events  are  obtained  from  data  con- 
I  cerning  the  number  and  grade  of  personnel  employed,  the  duration  of 
I  employment,  necessary  expenditures  for  equipment  and  facilities  and 
1  appropriate  allowances  for  overhead  assignments.  This  consideration 
I  of  the  elements  of  cost  is  in  keeping  with  conventional  budget  prepa- 
;  ration  practices.   What  is  different  is  the  relatively  tenuous  accuracy 
j  of  the  cost  estimate.   If  the  administrator,  in  the  preparation  of  the 

estimate,  has  improperly  assessed  any  one  of  several  technological 
'aspects  of  the  project,  the  effort  anticipated  in  the  funding  estimate 

may  be  entirely  inappropriate  or  inadequate  for  the  task. 

Using  the  events  of  the  scenario  as  milestones  of  progress, 

! 

it  is  possible  to  develop  cost-achievement  and  time-achievement 
functions  for  the  project.   Before  these  functions  can  be  expressed, 
a  measure  of  achievement  must  be  determined.   Methods  of  indicating 
achievement  are  numerous.  One  such  method,  in  terms  of  a^/a^j,  was 
discussed  earlier  in  this  chapter.   Due  to  the  diversity  of  activity 
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! 


that  is  experienced  during  the  course  of  a  project,  however,  a  single 
parameter  may  not  be  appropriate.   A  more  useful  measure  of  achievement 
for  measuring  project  progress  can  be  obtained  as  follows: 

Each  event  in  the  scenario  is  assigned  a  weight,  w. ,  repre- 
senting the  degree  of  difficulty  associated  with  the  event,  measured 
on  a  0  to  10  scale.  These  weights  are  then  multiplied  by  the  number 
'of  man-hours  assigned  to  corresponding  events,  h..   The  amount  of 

achievement  for  a  given  event,  a^,  is  then  equal  to  its  weighted 
i 
j number  of  man-hours  divided  by  the  total  weighted  man-hours.   That  »g  , 


wihi 


I"i*i 


With  achievement  thus  determined,  the  required  functions  can  then  be 
plotted  as  in  Figure  13. 

The  Theoretical  Model 


Having  obtained  time-achievement  and  cost-achievement  relation- 
ships, the  next  step  is  to  determine  how  they  may  be  used  in  project 
control.   If  such  a  determination  is  to  be  made,  it  will  be  necessary 
to  obtain  an  analytical  model  for  the  research  and  development  process. 

The  development  of  a  model  can  commence  with  an  analysis  of  the 
time  function.  To  begin,  it  is  recognized  that  the  time  for  completion 
of  any  event  is  a  random  variable.  The  estimated  value  of  time,  t-, 
will  rarely  be  the  actual  time  realized.  Whether  t-  is  greater  or 


Souder,  "Experiences  with  an  R  §  D  Project  Control  Model," 


p.  43. 
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FIGURE  13 
PROJECT  TIME  AND  COST  FUNCTIONS 


qp^pTp^ 


4 


Source:  Original 
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smaller  than  the  actual  t-  will  depend  on  the  ability  of  the  person 
making  the  estimate  to  anticipate  a  number  of  unknown  factors  in  the 
project.   These  factors  include  the  suitability  of  the  research  team 

to  the  task,  correct  assessment  of  technological  difficulties,  and  so 

j 
forth.  To  account  for  the  uncertainty  involved,  consider  a  hypo- 

I 
thetical  man-day  of  work*      This  quantity  is  a  Tayloristic  type  of 


measure  and  relates  to  the  amount  of  progress  that  can  be  realized 
I  by  a  man  who  is  ideally  suited  for  a  particular  task  and  devotes  100 
i  per  cent  of  his  time  to  that  task  for  one  working  day.  Assuming,  also, 
|  that  each  portion  of  a  project,  leading  to  an  event,  is  composed  of 
|  relatively  homogeneous  activity,  it  can  then  be  assumed  that  each 
j  event  is  composed  of  some  number  of  man-days  of  work.  Moreover,  this 
|  number  of  man-days  of  work  is  intrinsic  to  the  event  itself,  given 
|  the  state  of  the  project  from  the  preceding  event.  With  this  concept 
I  in  mind,  one  realizes  that  t^  is  art  estimate  of  some  t^,  the  number 
of  calendar  days  actually  required  to  accomplish  the  number  of  man-days 
of  work  associated  with  event  i. 

The  quantity  t.  can  be  examined  further  by  letting  k  be  the 
number  of  man-days  of  work  for  the  event.  Then,  t-  is  the  number  of 
calendar  days  that  will  pass  before  k  man-days  of  work  are  realized. 
Since  project  progress  is  not  constant,  the  number  of  calendar  days 
for  any  given  man- day  of  work  is  a  random  variable.  Assume,  then, 
that  there  is  a  probability  of  p  that  a  man-day  of  work  will  be 
accomplished  in  the  calendar  interval  At.   From  this,  one  observes 
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that  the  variable  t.  is  Poisson  distributed.  >  That  is, 


ti  -u 
P(tijy)  =  \^—  (Hi-21) 

where  y  is  the  mean  of  t^. 

The  Poisson  model  thus  derived  can  be  related  to  the  real 
world  by  examining  the  meaning  of  the  parameters  involved.   For  in- 
stance, k,  the  number  of  man-days  of  work,  will  be  a  function  of  the 

I 

i discipline  involved  (for  example  high-energy  physics  or  basic  mechan- 

j ics)  and  the  degree  of  achievement  to  be  realized  as  stated  in  the 

i 

|  project  objectives.  The  probability,  p,  and  mean,  \i,   will  be  functions 

j 

i  of  the  expertise  of  the  research  group  and  the  effectiveness  of  super- 

1 

2 
i vision.   In  retrospect,  then,  t  is  an  estimate  of  a  random  variable, 

-1    . 

i tj.   Errors  in  the  estimate;  or  tj-tj,  may  be  due  to  two  factors.  One 
j  factor  is  the  random  nature  of  t..  The  estimate,  at  best,  is  an 
I  estimate  of  the  mean,  p.  Another  factor  is  a  lack  of  competence  or 
the  presence  of  bias  on  the  part  of  the  person  making  the  estimate. 
After  determining  a  model  for  the  time  parameter  in  the  re- 
search and  development  process,  it  is  not  difficult  to  adopt  it  to  the 
cost  factor.   Recognizing  that  direct  costs  are  a  strong  function  of 
project  time,  one  can  see  that  each  man-day  of  work  will  relate  to  an 
intrinsic  cost.  Thus,  the  cost  of  a  project  will  also  be  Poisson 


Richard  C.  Dubes,  The  Theory  of  Applied  Probability  (Englewood 
Cliffs,  New  Jersey:   Prentice-Hall,  Inc.,  1968),  pp.  168-70. 

The  use  of  a  Poisson  model  was  suggested  by  Asher,  "A  Linear 
Programming  Model  for  the  Allocation  of  R  5  D  Efforts,"  p.  156. 
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distributed  and  can  be  expressed  by 


P(ci^)  =  iLTir-  (111-22) 


The  factors  affecting  the  accuracy  of  cost  estimates  are  the  same  as 
those  that  applied  to  the  tine  parameter. 

Analysis  and  Inference 

The  ultimate  objective  of  the  preceding  analysis  is  the  reali- 

i 

i zation  of  a  method  for  improving  project  control  techniques.   As  has 
j 
been  discussed  at  some  length,  most  control  techniques  are  based  on 

prior  estimates.   These  estimates,  however,  relate  to  processes  that 
|  are  inherently  uncertain  as  to  outcome.  What  is  needed,  then,  is  a 
■method  for  measuring  the  accuracy  of  prior  estimates  as  they  relate 
I  to  the  actual  situation. 

The  purpose  of  developing  a  scenario  is  to  obtain  time  and  cost 
I  estimates  for  a  project.   In  doing  this,  several  intermediate  estimates. 

for  scenario  events,  are  made.   In  considering  these  prior  estimates, 
I  the  manager  is  interested  in  knowing  whether  the  estimates  for  total 


time, 

. 

Ti '  -  I   ti 

(111-23) 

1 

and  total  cost, 

*i  ■  I  h 

(111-24) 

l 
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are  accurate  within  certain  bounds.  Most  control  methods  delay  con- 
clusions, regarding  estimate  accuracy,  until  the  final  stages  of  a 
project.  However,  if  the  mechanism  of  the  time  and  cost  processes  are 
understood,  such  conclusions  may  be  anticipated. 

The  errors  in  prior  estimates,  that  the  manager  needs  to 
identify,  are  those  resulting  from  incompetence  or  bias  on  the  part  of 
the  person  making  the  estimate.   If  this  person  is  of  a  conservative 

|  nature,  he  may  arbitrarily  introduce  some  amount  of  margin  in  all 

; 
: estimates  to  protect  himself  from  unanticipated  problems.   Or,  if  he 

I 

'  is  a  perpetual  optimist,  he  may  always  underestimate  cost,  or  over- 

i 

:  estimate  the  ability  of  the  research  team.   If  these  errors  in  the 

total  estimate  do  exist  (the  fact  that  is  to  be  determined)  it  is 

i  ■ 

!  reasonable  to  expect  that  proportionate  errors  will  exist  in  each  of 
I  the  estimates  for  the  events  composing  the  total.  Therefore,  as  the 
i  first  few  events  are  reached,  the  time  and  costs  actually  experienced 
j  should  be  useful  in  anticipating  the  ultimate  outcome.   Some  techniques 

i 

t 

i which  are  available  for  the  use  of  data,  obtained  subsequent  to  the 
prior  estimate,  can  be  illustrated  by  considering  the  following 
example:   Let  the  data  shown  in  Table  7  be  the  prior  estimates  of  cost, 
for  a  project  consisting  of  seven  scenario  events,  and  the  actual 

i 

costs  realized  for  the  first  four  events.   For  the  moment  though, 
assume  that  only  event  1  has  been  completed. 

After  the  occurence  of  event  1,  the  data  in  Table  7  is  related 


as 


c  =  10 
1 
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and     cj  =  14.0 

• 

TABLE  7 
PROJECT  COST 

DATA 

Event   Estimate 

Actual 

1 

10.0 

14.0 

..-- 

2 

14.0 

18.5 

3 

8.0 

12.3 

4 

24.0 

32.0 

5 

31.0 

— 

6 

12.0 

-- 

7 

9.0 

— 

The  objective  of  a 

control 

process,  at  th 

is  point, 

is  to  determine 

whether  the  error 

* 

c  -  cx  =  4.0  (111-25) 

1  I 

is  significant.  The  pertinent  facts  which  will  be  used,  in  determining 
significance,  are  that  the  random  variable,  c,,  is  Poisson  distributed, 
and  that  c^  is  the  estimate  of  the  mean,  u.  The  appropriate  hypothesis 
in  this  instance  is 
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! 

H0  :  cx  -  p  •<  10  . 

! 

-  Assuming  H  ,  the  test  to  be  conducted  is  to  determine  the  probability 

I  that  the  actual  results,  c*  would  occur  if 


which  is1 


14.0 
I         p(c,;10)  =  l-[   P(ci;l0)  (111-27) 

Cj-14.0  cx=0 

I  This  test  is  accomplished  with  the  use  of  the  cumulative  Poisson  dis- 

I 

j  tribution  which  is  tabulated  in  Appendix  B.   Using  Equation  (I 11-26) 

i 

I  one  finds 


I         p(ci;l0)  =  1  -  P(14.0;10) 


0^14.0 


-   1  -  0.917 


0.083 


If  a  confidence  limit  of 


a  =  .05 


had  been  adopted,  then,  since 


Ya-lun  Chou,  Statistical  Analysis  (New  York:  Holt,  Rinchart 


and  Winston,  Inc.,  1969),  p.  209, 


98 


.083  >_  a 

the  hypothesis,  H  ,  would  be  accepted.  This  amounts  to  accepting  the 
variation  of  Equation  (I I 1-25)  as  being  insignificant  due  to  the  ran- 
dom nature  of  the  variable  c.. 

As  more  data  becomes  available  with  accomplishment  of  other 
events,  as  shown  in  Table  7,  a  more  comprehensive  analysis  is  possible. 
j  The  type  of  analysis  most  commonly  employed,  with  several  samples 

J  from  similar  populations,  is  the  x  test.*  This  test  is  especially 

2 
j useful  in  testing  Poisson  distributed  data.   Application  of  the  x   test 

for  the  data  in  Table  7  is  illustrated  in  Table  8. 

2 
As  is  indicated  in  Table  8,  x  is  a  measure  of  the  sum  of  the 

squares  of  the  variates,  as  calulated  by 


x2  =  I    (ci  :  <H)   .  (111-28) 

i     ci 


2 
In  order  to  develop  a  X   analysis,  it  is  necessary  to  determine 

the  degrees  of  freedom  of  the  samples.  Without  pursuing  a  lengthy 

discussion  of  the  meaning  of  this  quantity,  suffice  it  to  say,  that,  in 

the  type  of  analysis  being  conducted,  when  n  samples  are  available, 

2 
there  will  be  n  degrees  of  freedom. 


1C.  G.  Paradine  and  B.  H.  P.  Rivett,  Statistical  Methods  for 


Technologists  (London:   The  English  Universities  Press, 

Ltd. 

,  1960) 

p.  79. 

2Ibid.,  pp.  76-77. 
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2 
The  results  of  a  x   test  are  obtained 

from  a  comparison  of  a 

computed  value  of  x  with  a  theoretical  value 

determined  on  the  basis 

of  the  degrees  of  freedom  and  the  confidence 

limit,  a.   Referring  to 

the  tabulation  in  Appendix  Q  it  is  seen  that 

for  four  degrees  of 

freedom  (v=4)  and  a=.05,  then  the  theoretical 

2 
value  of  x   is  9.49. 

Then,  since 

^  tci  "  cj)=  7j92  <_  9.49  (111-29) 


Ci 


one  would  conclude  that  the  variations,  c-  -  c • ,  are  within  the  limits 

of  error  attributed  to  the  random  nature  of  c. . 

i 

Control  Aspects  of  the  Analysis 

Both  of  the  statistical  tests,  conducted  in  the  preceding 
section,  resulted  in  the  conclusion  that  observed  errors  between 
estimated  and  actual  expenditures  were  attributable  to  random  varia- 
tion. However,  observing  the  data  in  Table  7,  one  is  struck  by  the 
fact  that  the  test  results  were  obtained  from  data  which  demonstrated 
cost  overruns  of  up  to  50  per  cent.   It  is  natural,  then,  to  question 
the  usefulness  of  these  methods.   However,  certain  factors  in  the 
analysis  should  be  examined. 

First,  it  is  necessary  to  recognize  the  statistical  character 
of  the  Poisson  distribution.   For  instance,  the  distribution  has  a 
pronounced  skewness  that  is  a  function  of  the  value  of  the  random  var- 
iable.  In  addition  to  this,  the  variance  is  equal  to  the  mean,  which 
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generally  results  in  substantial  dispersion.   The  end  result  of  these 
characteristics  is  that  statistical  tests  based  on  Poisson  data  are 
oftcr  less  precise  than  tests  of  normally  distributed  data,  for  ex- 

I  i 

ample.   From  this,  one  can  conclude  that  conventional  confidence  limits 

\  i 

;  (for  example  a=.05  or  a=.01),  which  are  generally  associated  with  nor- 
•  raal  data,  may  be  insufficient  constraints  for  tests  of  Poisson  data, 
j For  example,  the  selection  of  a=.10  vice  .05  would  have  changed  the 
j conclusions  drawn  from  the  example  tests. 

A  second  important  consideration,  which  must  be  recognized,  re-  I 

j  lates  to  the  model  whicli  was  developed.   It  must  be  recalled  that  the 

I 

-,  data  and  tests  which  were  conducted  were  drawn  from  a  model  which  cor- 

! 

!  related,  for  instance,  cost  and  technological  achievement.   It  must 

;  l 

!  further  be  recognized  that  overruns  in  research  expenditures  frequently 
j f ar  exceed  the  50  per  cent  level.   At  this  point,  one  might  wonder 
;  about  research  and  development  projects  which,  after  demonstrating  ex- 
ices  si  ve  costs  in  early  stages,  are  constrained  to  complete  within  final 
[budget  figures.  This  does  frequently  occur.  However,  what  invariably 

i  accompanies  such  forced  adherence  to  budget  limitations  is  a  concomi- 

I 

;  tant  revision  of  project  objectives.   Consequently,  though  expendi- 

|  ! 

ture  estimates  may  ultimately  be  proven  to  be  quite  accurate,  consider-  ! 

j  i 

:  able  shortcomings  will  probably  be  realized  in  estimates  of  achieve- 

i 

j  ment . 

These  considerations  certainly  are  significant  in  any  discus- 

'  sion  of  the  usefulness  of  the  statistical  techniques  that  have  been 

developed.   It  is  fairly  obvious  that  two  conditions  must  be  satisfied 

;  if  such  inferential  analysis  is  to  be  useful.  Any  organization  which 
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might  consider  such  techniques  must  first  of  all  identify  a  realistic 
confidence  limit,  in  keeping  with  what  are  considered  to  be  acceptable 
amounts  of  over- run.   Second,  it  is  necessary  to  ensure  that  achieve- 
ment parameters  associated  with  project  events  are  valid  measures  of 
technological  advance.   If  this  is  assured,  it  must  then  be  accepted 
that  attempts  to  enforce  budgetary  requirements  will  likely  result  in 
a  failure  to  achieve  intended  project  objectives. 

Statistical  techniques,  such  as  those  examined,  do  offer  add- 
itional methods  for  controlling  research  and  development.   By  conduct- 
ing periodic  review  which  incorporates  these  techniques,  additional 
information  can  be  obtained  which  is  pertinent  to  project  control. 
However,  action  resulting  from  such  review,  whether  it  be  to  continue, 
terminate,  re-evaluate  or  whatever,  must  be  based  on  an  understanding 

of  what  is  actually  indicated  by  the  results  of  the  analysis. 

! 

Summary 

j 
Chapter  II  was  devoted  to  the  description  of  numerous  methods 

which  have  been  used  or  have  application  in  project  selection  and  con- 
trol. This  chapter  has  addressed  the  problem  of  selecting  particular 
techniques,  from  among  those  available,  which  are  considered  especial- 
ly useful  in  the  selection  process.   In  addition,  a  statistical  control 

technique,  not  found  elsewhere  in  the  literature,  has  been  developed, 

i 

and  its  applicability  in  project  control  has  been  discussed.  Though 

i 

the  several  ingredients »  of  the  selection  and  control  techniques  that 
have  been  developed,  have  been  drawn  from  other  sources,  it  is  con- 
sidered that  the  organization  of  these  ingredients  that  has  been  given 

i 
in  this  analysis  constitutes  a  new  approach  to  project  selection  and 
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I : 1 

icontrol.   The  procedure  for  implementing  the  selection  technique  has 
ibeen  adequately  summarized  in  Figure  11.   This  technique  has  emphasized 
'the  use  of  information  that  is  generally  available  in  research  and 

I 

idevelopment  administrative  efforts.   It  has  minimized  the  use  of  data 
jwhich  might  require  special  collection  efforts.   In  particular,  it  has 
j avoided  the  use  of  estimates  of  probabilities  of  success. 

The  development  of  the  statistical  technique  for  use  in  project 

control  has  addressed  only  the  basic  theoretical  concept  which  forms 

its  foundation  and  the  demonstration  of  its  use.   Actual  application  of 

| the  technique  has  not  been  pursued  because  of  the  variety  of  methods  in 

I  which  it  might  be  used.   As  has  been  emphasized,  the  frequent  exclusive 

! 

reliance  on  prior  estimates  for  project  control  is  fraught  with  un- 
i 
I 

certainty.  Yet,  prior  estimates  arc  an  essential  part  of  any  control 

; system.   The  statistical  technique  developed  herein  is  offered  as  an 
| adjunct  to  existing  control  methods.   It  is  considered  that  this  tech- 
nique, when  combined  with  usual  management  control  practices,  will  im- 
i 
prove  the  control  of  research  and  development  programs.  This  technique 

should  enhance  management  by  exception  in  that  it  offers  more  objective 
data  for  use.  This  technique  is  deemed  to  be  particularly  desireable 
because  of  the  manner  in  which  project  achievement  is  necessarily  in- 
corporated in  the  analysis.   Presumably,  this  will  serve  to  minimize 
the  need  for  presumptions  regarding  progress  that  are  usually  drawn 
from  an  examination  of  time  and  cost  accounting  data. 

The  theoretical  treatment  that  has  been  offered  in  this  chap- 
ter is  noticeably  not  rigorous.  The  reason  for  this  is  that  the  com- 
ponent methods  of  the  techniques  that  have  been  developed  were  drawn 
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from  other  sources  and  are  presumed  proven.   Emphasis  has  therefore 
been  placed  on  the  organization  of  these  components  into  a  useful, 
coherent  system. 


CHAPTER  IV 


CONCLUSIONS 


This  paper  has  attempted  to  present  an  analysis  of  research 

and  development  practices  with  the  intent  of  producing  techniques  which 

i 
should  be  useful  in  efforts  associated  with  project  selection  and  con- 


jtrol.   This  effort  was  prompted  by  notable  shortcomings  in  existing 

I  management  methods  as  evidenced  by  widely  publicized  failures  in  major 

!  programs.   As  has  been  discussed  in  the  preceding  chapters,  several 

|  problem  areas  can  be  identified  as  being  likely  candidates  for  the 

;  cause  of  the  difficulties  that  have  been  experienced.   Foremost  among 

these  is  the  inadequacy  of  data  employed  in  present  systems. 

Most  contemporary  efforts  in  the  management  of  research  and 

!  development  rely  on  the  use  of  data  adopted  from  established  production- 

,;  type  operations.  These  data,  invariably,  emphasize  the  use  of  time  and 

>  cost  accounting.   However,  as  has  been  forcibly  asserted  throughout 
I 

the  preceding,  this  reliance  on  conventional  methods  is  inadequate; 
| 

:  due  to  the  nature  of  research  and  development  activities.   One  should 
j 

!  be  able  to  conclude,  by  now,  that  controlling  by  budget  overruns  and 

I 

;  underruns  only,  can  be  seriously  misleading.   It  must  be  emphasized 

;  that  it  is  the  interaction  of  both  cost  and  achievement  that  determines 

j 

;  the  actual  status  of  a  research  project.  The  conclusion  that  must  be 

I  drawn  from  this  is  that  existing  methods  of  controlling  projects  on 

; 

\   the  basis  of  budgetary  variances,  which  are  time  and  cost  functions 

105 
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i 
alone,  are  exercises  in  futility  and,  at  best,  will  only  give  the  man- 
ager a  deceptive  feeling  of  satisfaction  concerning  the  conduct  of  a 
project.  This  deception  is  realized  only  in  the  latter  stages  of  a 
project  when  it  is  necessary  to  accept  the  fact  that  objectives  will 
not  be  accomplished  without  additional  expenditures.   It  is  at  this 

point,  when  most  of  the  available  funds  have  already  been  spent,  that 

i 
the  manager  feels  compelled  to  spend  more  in  order  to  avoid  criticism 

for  having  apparently  wasted  preceding  expenditures.  The  only  method 

available  to  alleviate  such  an  "eleventh-hour"  panic  is  to  continually 
|  ! 

measure  progress  throughout  the  life  of  the  project.  The  effectiveness 

i 
of  such  monitoring  will,  of  course,  depend  on  the  efficiency  of  any 

analyses  of  variances  which  do  occur. 

i 
In  an  attempt  to  account  for  the  factors  of  uncertainty 'which 

!  i 

i 
are  frequently  encountered,  some  techniques  have  relied  on  a  measure 

! 

of  the  probability  of  success  of  subject  projects.   It  should  be 

j 
patently  obvious  that  reliance  on  such  a  device  pre-empts  any  efforts 

;to  monitor  subsequent  performance.  The  probability  of  success  measure 
'  i 

■  per  force  assumes  that  any  determination  regards  success  must  be  re- 
served until  the  conclusion  of  a  project.   By  adopting  this  technique, 

;  the  manager  neglects  his  responsibility  to  assess,  in  a  continuing 

I 

.  fashion,  the  successful  progress  of  a  project. 

These  shortcomings  would  necessarily ' lead  one  to  conclude  that 
existing  methods  for  the  management  of  research  and  development  are  not 
sufficient  for  their  intended  purposes.   The  reasons  that  more  fail- 
ures than  those  experienced  have  not  occurred  are  that  management  has 
been  quite  fortunate  or  research  objectives  have  been  sacrificed  to 
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avoid  overruns. 

The  techniques  for  project  selection  and  control  that  have 
been  presented  in  this  paper,  specifically  in  Chapter  III,  are  regarded 
as  offering  at  least  a  partial  solution  for  the  difficulties  that  have 
been  discussed.  The  selection  technique  demands  little  in  terms  of  the 

effort  required  for  implementation.  The  theory  is  straight-f orward  and 

i 
i 
j the  assumptions  are  obvious.   The  very  simplicity  of  the  method  will 

likely  invite  criticism  that  such  simplicity  cannot  possibly  account 

i 

j f or  the  numerous  intricacies  of  research  and  development  projects. 

:  Such  criticism  cannot  be  denied.   Indeed,  it  has  been  a  major  premise 

of  this  paper  that  such  intricacies  are  of  paramount  importance  and 

■cannot  be  accounted  for  in  any  selection  process,  regardless  of  the 
i 
degree  of  sophistication  that  might  be  offered.  However,  it  must  be 

;  acknowledged  that  the  proposed  selection  technique  is  intended  for  use 
with  the  control  technique  which  was  the  second  subject  of  the  pre- 

|  ceding  chapter. 

It  has  been  stressed  that  any  selection  method  must  be  based 
on  prior  estimates.   However,  efforts  to  refine  such  methods  of  se- 

! 

lection  cannot  achieve  a  degree  of  certainty  beyond  the  limits  imposed 
by  the  inherent  nature  of  the  projects  being  selected!   Therefore, 
instead  of  continuing  to  emphasize  the  importance  of  selection,  one 
must  begin  to  develop  methods  to  monitor  progress  so  that  selection 
errors  can  be  identified  in  a  timely  fashion.   Selection,  then,  must 
be  regarded  as  only  a  portion  of  a  system  for  project  management.  The 
other  very  important  portion  of  the  system  is,  of  course,  control. 
The  development  and  implementation  of  an  effective  control 
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technique,  relieves,  somewhat,  the  burden  placed  on  selection.   If 

effective,  control  will  identify  errors  that  have  been  made  in  selection 

i 
| so  that  funds  can  be  diverted  to  more  successful  endeavors.   Thus, 

the  development  of  an  effective  control  system  is  possibly  the  best 

means  of  accounting  for  uncertainty  in  research  and  development. 

! 
A  final  aspect  of  statistical  control,  that  deserves  mention, 

I 
lis  that  of  credibility.  As  was  shown  in  the  preceding  chapter,  it  may, 

at  times  be  difficult  to  accept  the  results  of  analysis.  This  is, 

however,  a  difficulty  that  is  fairly  common  among  Baysian-style  methods. 

It  can  be  said,  in  this  respect,  that  such  techniques  are  useful  only 

when  implemented  in  an  adequate  fashion.  Attempts  to  institute  such 

systems  on  a  partial  or  "trial"  basis  seldom  prove  satisfactory  since 

subjective  influences  invariably  offset  the  prescriptions  of  objective 

i  i 

techniques. 

These  prospects  of  difficulties  in  implementation  prompt  con- 
i 
sideration  of  certain  aspects  of  the  proposed  techniques  which  deserve 

,  further  examination  and  refinement.  The  first  of  these  is  the  need  to 

! 

I  develop  a  uniformly  most  powerful  test  for  the  Poisson  model.  The  tests 

[proposed  in  this  paper  are  valid  for  any  situation;  however,  in  some 

i  circumstances  -they  may  be  found  lacking  in  precision.   It  would  there- 
l 

I  fore  be  advantageous  to  find  a  method  for  structuring  data  such  that 

! 

the  tests  will  yield  consistent  results  for  any  circumstances. 

A  second  consideration  of  the  model  which  merits  further  de- 

I 
velopment  is  the  need  to  ensure  accurate  portrayal  of  technological 

progress  in  project  scenarios.  As  has  been  discussed  in  preceding 

sections,  the  measurement  of  progress  in  research  activities  is  often 
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I 
quite  difficult.   However,  for  the  techniques  proposed  it  is  absolutely 

essential  that  events  be  selected  such  that  progress  is  explicitly  in- 

I 
dicated.   Procedures  for  identifying  adequate  indicators  and  computing 

I  ! 

[progress  need  further  development  in  order  that  this  requrement  can  be 
satisfied. 

A  final  aspect  which  needs  more  refinement  concerns  the  optimi- 
zation method  that  has  been  selected.  To  be  specific,  this  method  does 
not  allow  for  minimal  funding  of  projects.  In  many  instances,  though  a 
project  may  not  be  included  in  an  optimal  funding  program,  management 
may  wish  to  provide  some  token  appropriation  in  order  that  project  con- 
tinuity is  not  lost  completely.  For  the  proposed  technique,  though,  a 
project  is  either  funded  at  an  adequate  level  or  not  at  all.   It  there- 

ffore  would  be  worthwhile  to  formulate  a  selection  technique  which  will 
optimize  allocations  with  the  provision  that  a  certain  number  of  non- 
optimum  projects  may  receive  some  funding. 


APPENDIX  A 
MISSION-TECHNOLOGY  VALUE  MATRIX1 
An  essential  part  of  the  selection  process  is  the  identifica- 

I 

tion  of  the  contribution  of  technologies  to  known  objectives.   These 
matrices  were  developed  for  the  purpose  of  relating  technologies, 

supported  by  Naval  Research  efforts,  to  the  research  objective  areas  of 

I 
the  Naval  Materiel  Command,  Washington,  D.C.   These  matrices  are  in- 

! 

eluded  to  illustrate  the  degree  of  detail  required  in  identifying  all 
pertinent  relationships.  Counterparts  to  these  matrices  would  be  sim- 
ilar ones  which  relate  sciences  to  the  technologies  given. 


1Marvin  J.  Cetron,  "QUEST  Status  Report,"  IEEE  Transactions 
on  Engineering  Management,  EM-14  (March,  1967),  p.  58. 
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STRIKE  WARFARE 

Airborne  Attack 
Surface  Attack 
Submarine  Attack 
Amphib .   Assault 
Seabased  Strategic 

Deterrence 
Airborne  Anti-Air 

Warfare 
Surface  Anti-Air 

Warfare 

AIITI-SUB  WARFARE 
Airborne  ASW 
Surface  AS1/: 
Submarine  ASW 
Undersea  Surv. 
Mining 

Mine   Cntr-meas, 
ASW  Support 

COMMAND  SUPPORT 
Command  &  Control 
Naval  Communicat. 
Electr.  Warfare 
Navigation 
Ocean  Surv. 
P.econ.  i:  Intel!. 
Environmental 

Systems 
Special  Warfare 
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APPENDIX  B 
CUMULATIVE  POISSON  DISTRIBUTION1 


A  tabulation  of 


x 


F(x)  =  P(c<x)  =  I     e"xXc/c! 
c=0 


^ Bernard  Ostle,  Statistics  in  Research  (Ames,  Iowa:  The 
Iowa  State  University  Press,  1963),  pp.  372-76. 
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APPENDIX  C 
CHI-SQUARE  DISTRIBUTION1 
A  tabulation  of  the  probability  that  x2  will  be  exceeded  (a). 

v\a        .995        .990        .975        .950        .050        .025        .010        .005 

1  --  --  --  .004      3.84        5.02        6.63        7.88 

2  .01  .02  .05        .A0        5.99        7.38        9.21      10.60 

I 

3  .07  .11  .22  .35        7.81        9.35      11.34      12.84 

I 

4  .21  .30  .48  .71        9.49      11.14      13.28      14.86 

I 

5  .41  .55  .83        1.15      11.07     12.83     15.09      16.75 

6  .68  .87        1.24        1.64      12.59      14.45      16.81      18.55 

i 

7  .99        1.24        1.69        2.17     14.07     16.01      18.48     20.28 

8  1.34        1.65        2.18        2.73     15.51      17.53     20.09     21.96 

9  1.73        2.09        2.70        3.33      16.92      19.02      21.67      23.59 
10              2.16        2.56        3.25        3.94      18.31      20.48      23.21      25.19 


*Chou,  Statistical  Analysis,  p.  775, 
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